Hi Selva, Adding the coprocessor as an attribute of the table is a good plan. That would prevent the case where a coprocessor is not loaded at all for a table, which is what I was concerned about. Thanks for clarifying this issue for me!
I am still not clear how software migration would occur. Would disabling and enabling the tables work in your proposal? If not, how can we guarantee a clean exit of one version of coprocessor code before starting up with another? Thanks, Sean -----Original Message----- From: Selva Govindarajan [mailto:selva.govindara...@esgyn.com] Sent: Tuesday, October 25, 2016 9:11 AM To: dev@trafodion.incubator.apache.org Subject: RE: [DISCUSS] Loading coprocessors from the client side dynamically Sean, My proposal and the implementation of PR 777 is to make Trafodion behave consistently irrespective of distro and the type of installation. Trafodion will always add table coprocessors now so that region server doesn't call into Trafodion hooks for non-trafodion tables. In addition, the proposal is to have a resource to customize Trafodion instead of using the standard hbase-site.xml. Though the PR 777 doesn't remove the 'hbase.coprocessor.region.classes' from the hbase-site.xml from the cluster installer, it would be done sometime later. This shouldn't prevent any recovery process. The first attempt is to try this out on workstation before implementing it on cluster install. For the existing Trafodion installations, all the trafodion tables should be altered to add this coprocessor via hbase shell, before the above property is removed from hbase-site.xml Do you anticipate any issues with this? By adding table coprocessor, Trafodion needs to stick with these coprocessor classes. If there is a need to add more coprocessor classes, it should be added to all the existing tables manually or it should be designed such that Trafodion can tolerate the absence of the newly added coprocessors. If there is a need to avoid restarting hbase, it might require extensive testing and at least the following needs to be done: 1) Trafodion shouldn't require the modifications of ACL of some of the hdfs directories for the fresh Trafodion cluster installation. 2) Assuming region server supports class reloading concept for the table coprocessors, Trafodion needs to adapt to it for its upgrade. Selva -----Original Message----- From: Sean Broeder [mailto:sean.broe...@esgyn.com] Sent: Tuesday, October 25, 2016 3:00 AM To: dev@trafodion.incubator.apache.org Subject: RE: [DISCUSS] Loading coprocessors from the client side dynamically This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing Selva, I wasn't thinking of a particular pull request when I sent this out for discussion. I was just thinking in general because there was the hope of being able to install Trafodion without restarting HBase. Regarding the specific pull request you mentioned, I see no issue because it only affects install_local_hadoop and that explicitly does an initialize Trafodion, so there is no recovery to worry about. This should be fine. Also, I have no issue with configuring a coprocessor as a table attribute. This certainly loads the coprocessors. It's unclear for me if there is still an intent to avoid HBase restart or not. If not, then this discussion is mute. But if there is, can you tell me how the software migration from version X to version y would work? If you change the class path to point away from version x and toward version y, then disable and re-enable each table in order to close and reopen all the regions, this might work. I think you'd want to lock the database to prevent transactions during this time. Am I understanding your proposal correctly? Thanks, Sean -----Original Message----- From: Selva Govindarajan [mailto:selva.govindara...@esgyn.com] Sent: Monday, October 24, 2016 10:37 PM To: dev@trafodion.incubator.apache.org Subject: RE: [DISCUSS] Loading coprocessors from the client side dynamically The coprocessors are neither pushed nor loaded dynamically from the SQL client side. They are simply added as the table attribute and hence configured as table coprocessor at the time of creation via Trafodion. So, if you do describe in hbase shell, you will see something like below describe 'TRAFODION._MD_.OBJECTS' Table TRAFODION._MD_.OBJECTS is ENABLED TRAFODION._MD_.OBJECTS, {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver|1073741823|', coprocesso r$2 => '|org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint|1073741823|', coprocessor$3 => '|org.apache.hadoop.hbase.coprocessor.AggregateIm plementation|1073741823|'} COLUMN FAMILIES DESCRIPTION {NAME => '#1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_V ERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} {NAME => 'mt_', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '2', TTL => 'FOREVER', MIN_ VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'true', BLOCKCACHE => 'true'} By configuring it as table coprocessor, the region server would dynamically load the needed classes and invoke the corresponding hooks when the table/region is operated on by it. Trafodion has started adding table coprocessor to its table possibly in Trafodion 1.0 release. However, it was added in CDH cluster installation wherein the property 'hbase.coprocessor.region.classes' is not visible in the client side of hbase. It doesn't happen in case of Hadoop installation via install_local_hadoop for the developer workspaces. At the time of adding of table coprocessor it was thought we can avoid hbase restart by doing it. It was soon found that restart can't be avoided. However, adding table coprocessor has another advantage that coprocessor classes hooks are invoked for trafodion tables only. It is not invoked for non-trafodion tables. For the 2 main points in the discussion: The table coprocessor attribute just has the fully qualified class name.. The region server needs to find this class from its own classpath. Hence it is not clear how incompatible version of software would exist? Is it because the jars containing these classes from the compatible version are not copied to RS class path locations? If so, the same problem would exist with/without table coprocessors? Do you think TrxRegionEndPoint and TrxRegionObserver wouldn't be loaded if it is configured as table coprocessor attribute only? The PR 777 doesn't remove the property 'hbase.coprocessor.region.classes' from hbase-site.xml for cluster install. It is removed from install_local_hadoop. I was able to run full regressions without the property 'hbase.coprocessor.region.classes' in hbase-site.xml? Is there a way to simulate the condition that you had described so that I can verify if it works as expected or as feared? Selva -----Original Message----- From: Sean Broeder [mailto:sean.broe...@esgyn.com] Sent: Monday, October 24, 2016 7:59 PM To: dev@trafodion.incubator.apache.org<mailto:dev@trafodion.incubator.apache.org> Subject: [DISCUSS] Loading coprocessors from the client side dynamically This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing Hi, I have seen some discussion related to pushing/loading coprocessors in the regions dynamically from the SQL client side when Trafodion is started or when a table is opened. The motivation here is to be able to install Trafodion without having to stop/restart HBase where a customer might have a previous HBase installation. I want to point out that Trafodion uses a TrxRegionEndpoint and a TrxRegionObserver coprocessor. These two coprocessors work in tandem to ensure region split, region rebalance, and region recovery are possible for multi-table transactions. These two coprocessors ensure data consistency and form the basis of our ACID transaction implementation. We currently mandate that these two coprocessors have entries in the hbase-site.xml file. While I have no problem loading other coprocessors dynamically, I do not think it is a good idea to load these two on the fly without stopping/restarting HBase, at least not in the current design. The issue is that other components in the DTM may have code changes that assume the corresponding versions of these coprocessors are loaded and running. If the DTM components are not compatible with the coprocessors, unpredictable results could occur, including system hangs, process failures, and data corruption. Furthermore, if the coprocessors are loaded dynamically and the entries are not in the hbase-site.xml file, then Trafodion recovery would effectively be abandoned before the regions are started and there would be no way to correct the inconsistencies short of dropping and recreating the tables. So the 2 main points I hope the readers of this discussion will take away based on the current Trafodion design are: 1) The TrxRegionEndpoint and TrxRegionObserver coprocessors must remain in the hbase-site.xml file to ensure recovery occurs before the regions are started and open for HBase activity. 2) Trafodion and the above coprocessors must run compatible versions of software to ensure proper function. These two coprocessors are not static, so it is likely that if Trafodion code has changed, so have the coprocessors. Thanks, Sean
