[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128550#comment-15128550
 ] 

ASF GitHub Bot commented on TRAFODION-1729:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-trafodion/pull/258


> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  
> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
> hbase.snapshot.enabled true
> hbase.bulkload.staging.dir/hbase-staging
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101435#comment-15101435
 ] 

ASF GitHub Bot commented on TRAFODION-1729:
---

Github user mashengchen closed the pull request at:

https://github.com/apache/incubator-trafodion/pull/258


> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  
> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
> hbase.snapshot.enabled true
> hbase.bulkload.staging.dir/hbase-staging
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101519#comment-15101519
 ] 

ASF GitHub Bot commented on TRAFODION-1729:
---

GitHub user mashengchen reopened a pull request:

https://github.com/apache/incubator-trafodion/pull/258

TRAFODION-1729

change the coprocessor deployment method, to avoid intrusive into 
underlying HBase system.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mashengchen/incubator-trafodion master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-trafodion/pull/258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #258


commit 12d7342bb3d63991d31933139f5b3fee216ab8b7
Author: mashengchen 
Date:   2015-12-14T06:16:25Z

Merge pull request #1 from apache/master

update head

commit 3450fb15d4ae86c20c0a31f7e0750709f433af59
Author: mashengchen 
Date:   2015-12-21T05:26:00Z

Merge pull request #2 from apache/master

update head

commit c8fef619f858ff24b8076e4abc2f556d9fb42acb
Author: mashengchen 
Date:   2015-12-23T13:09:41Z

Merge pull request #3 from apache/master

update

commit 0042344cfb67900a1c815da932e5ea4c7ba93a19
Author: mashengchen 
Date:   2015-12-30T10:32:40Z

Merge pull request #4 from apache/master

update head

commit 11f9686b5df87f70b79edac5621da0610c7e70b8
Author: mashengchen 
Date:   2016-01-11T02:49:44Z

jira1729

commit cac96bc7ea69a3f1293e9ddef5e752f8785fb886
Author: mashengchen 
Date:   2016-01-11T03:19:05Z

Merge pull request #5 from apache/master

update head

commit 62237083f0db8230e257b6a8a12b5ca21c3831d8
Author: mashengchen 
Date:   2016-01-11T04:03:24Z

Merge branch 'master' of github.com:apache/incubator-trafodion

commit 5292fdc470db2317f4ab1133d298e98b97992e1e
Author: mashengchen 
Date:   2016-01-11T04:19:31Z

Merge branch 'master' of github.com:mashengchen/incubator-trafodion

commit 0ea8552120bd12d8c5ed1757325276cca3eac172
Author: mashengchen 
Date:   2016-01-11T05:39:43Z

Update HBaseClient.java

remove unnecessary log.

commit 6e0193a1837148fe5fecd3653402ddaff61f54ce
Author: mashengchen 
Date:   2016-01-11T05:41:00Z

Update CoprocessorUtils.java

change log level

commit 0dbaf3b3164069587a854968869211d1b69c3434
Author: mashengchen 
Date:   2016-01-11T05:52:45Z

jira1729

commit d8b594f2b585de224a007e48ba735f0f5db860be
Author: mashengchen 
Date:   2016-01-12T04:15:12Z

add copyright

commit e1af6b9b94b237ab98c8e40677ada757a9b76f18
Author: mashengchen 
Date:   2016-01-12T05:38:26Z

rename & update code

commit 411b703f557bdcff63b971261715d9a2caecb989
Author: mashengchen 
Date:   2016-01-12T05:50:13Z

Merge branch 'master' of github.com:apache/incubator-trafodion

commit c1b8448c90533a5a2b039cce116ee57563c77299
Author: mashengchen 
Date:   2016-01-12T05:54:31Z

remove trafcoprocess.properties

commit b8e6e9c920039c2fd7c0a175214b93ed2335d3e9
Author: mashengchen 
Date:   2016-01-14T07:54:08Z

Update traf_coprocessor.properties

use TrxRegionEndpoint as default

commit b01d8cb00abcfcadf48125649bdb69aac8616ffd
Author: mashengchen 
Date:   2016-01-15T04:16:02Z

Update CoprocessorUtils.java

commit a005ebf5ccf75a374f034f8b63891a396c651ae9
Author: mashengchen 
Date:   2016-01-15T06:18:14Z

Update HBaseClient.java

change CoprocessorUtils.addCoprocessor, add isMVCC as param

commit d11e9d4ecf4263b7999d4bf8e6460ba78da04150
Author: mashengchen 
Date:   2016-01-15T06:22:24Z

Update CoprocessorUtils.java

change addCoprocessor add isMVCC as param.

commit 5751399d9334b77bbaca629b9c93d55355b49f00
Author: mashengchen 
Date:   2016-01-15T06:27:01Z

Update traf_coprocessor.properties

make mvcc & sscc as a separate ones

commit 379f8aefaf12476b8bc0f7960636a33b1b341449
Author: mashengchen 
Date:   2016-01-15T06:28:45Z

Update CoprocessorUtils.java

commit c15226c639802578bc6eeac38b27aadf93aac8fe
Author: mashengchen 
Date:   2016-01-15T06:36:35Z

Update CoprocessorUtils.java

commit accdfe2848fecfdfb1521a2162f11c65794f391d
Author: mashengchen 
Date:   2016-01-15T09:08:49Z

seems version conflect




> change the 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101314#comment-15101314
 ] 

ASF GitHub Bot commented on TRAFODION-1729:
---

GitHub user mashengchen reopened a pull request:

https://github.com/apache/incubator-trafodion/pull/258

TRAFODION-1729

change the coprocessor deployment method, to avoid intrusive into 
underlying HBase system.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mashengchen/incubator-trafodion master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-trafodion/pull/258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #258


commit 12d7342bb3d63991d31933139f5b3fee216ab8b7
Author: mashengchen 
Date:   2015-12-14T06:16:25Z

Merge pull request #1 from apache/master

update head

commit 3450fb15d4ae86c20c0a31f7e0750709f433af59
Author: mashengchen 
Date:   2015-12-21T05:26:00Z

Merge pull request #2 from apache/master

update head

commit c8fef619f858ff24b8076e4abc2f556d9fb42acb
Author: mashengchen 
Date:   2015-12-23T13:09:41Z

Merge pull request #3 from apache/master

update

commit 0042344cfb67900a1c815da932e5ea4c7ba93a19
Author: mashengchen 
Date:   2015-12-30T10:32:40Z

Merge pull request #4 from apache/master

update head

commit 11f9686b5df87f70b79edac5621da0610c7e70b8
Author: mashengchen 
Date:   2016-01-11T02:49:44Z

jira1729

commit cac96bc7ea69a3f1293e9ddef5e752f8785fb886
Author: mashengchen 
Date:   2016-01-11T03:19:05Z

Merge pull request #5 from apache/master

update head

commit 62237083f0db8230e257b6a8a12b5ca21c3831d8
Author: mashengchen 
Date:   2016-01-11T04:03:24Z

Merge branch 'master' of github.com:apache/incubator-trafodion

commit 5292fdc470db2317f4ab1133d298e98b97992e1e
Author: mashengchen 
Date:   2016-01-11T04:19:31Z

Merge branch 'master' of github.com:mashengchen/incubator-trafodion

commit 0ea8552120bd12d8c5ed1757325276cca3eac172
Author: mashengchen 
Date:   2016-01-11T05:39:43Z

Update HBaseClient.java

remove unnecessary log.

commit 6e0193a1837148fe5fecd3653402ddaff61f54ce
Author: mashengchen 
Date:   2016-01-11T05:41:00Z

Update CoprocessorUtils.java

change log level

commit 0dbaf3b3164069587a854968869211d1b69c3434
Author: mashengchen 
Date:   2016-01-11T05:52:45Z

jira1729

commit d8b594f2b585de224a007e48ba735f0f5db860be
Author: mashengchen 
Date:   2016-01-12T04:15:12Z

add copyright

commit e1af6b9b94b237ab98c8e40677ada757a9b76f18
Author: mashengchen 
Date:   2016-01-12T05:38:26Z

rename & update code

commit 411b703f557bdcff63b971261715d9a2caecb989
Author: mashengchen 
Date:   2016-01-12T05:50:13Z

Merge branch 'master' of github.com:apache/incubator-trafodion

commit c1b8448c90533a5a2b039cce116ee57563c77299
Author: mashengchen 
Date:   2016-01-12T05:54:31Z

remove trafcoprocess.properties

commit b8e6e9c920039c2fd7c0a175214b93ed2335d3e9
Author: mashengchen 
Date:   2016-01-14T07:54:08Z

Update traf_coprocessor.properties

use TrxRegionEndpoint as default

commit b01d8cb00abcfcadf48125649bdb69aac8616ffd
Author: mashengchen 
Date:   2016-01-15T04:16:02Z

Update CoprocessorUtils.java

commit a005ebf5ccf75a374f034f8b63891a396c651ae9
Author: mashengchen 
Date:   2016-01-15T06:18:14Z

Update HBaseClient.java

change CoprocessorUtils.addCoprocessor, add isMVCC as param

commit d11e9d4ecf4263b7999d4bf8e6460ba78da04150
Author: mashengchen 
Date:   2016-01-15T06:22:24Z

Update CoprocessorUtils.java

change addCoprocessor add isMVCC as param.

commit 5751399d9334b77bbaca629b9c93d55355b49f00
Author: mashengchen 
Date:   2016-01-15T06:27:01Z

Update traf_coprocessor.properties

make mvcc & sscc as a separate ones

commit 379f8aefaf12476b8bc0f7960636a33b1b341449
Author: mashengchen 
Date:   2016-01-15T06:28:45Z

Update CoprocessorUtils.java




> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091475#comment-15091475
 ] 

ASF GitHub Bot commented on TRAFODION-1729:
---

Github user mashengchen closed the pull request at:

https://github.com/apache/incubator-trafodion/pull/258


> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  
> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
> hbase.snapshot.enabled true
> hbase.bulkload.staging.dir/hbase-staging
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091477#comment-15091477
 ] 

ASF GitHub Bot commented on TRAFODION-1729:
---

GitHub user mashengchen reopened a pull request:

https://github.com/apache/incubator-trafodion/pull/258

TRAFODION-1729

change the coprocessor deployment method, to avoid intrusive into 
underlying HBase system.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mashengchen/incubator-trafodion master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-trafodion/pull/258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #258


commit 12d7342bb3d63991d31933139f5b3fee216ab8b7
Author: mashengchen 
Date:   2015-12-14T06:16:25Z

Merge pull request #1 from apache/master

update head

commit 3450fb15d4ae86c20c0a31f7e0750709f433af59
Author: mashengchen 
Date:   2015-12-21T05:26:00Z

Merge pull request #2 from apache/master

update head

commit c8fef619f858ff24b8076e4abc2f556d9fb42acb
Author: mashengchen 
Date:   2015-12-23T13:09:41Z

Merge pull request #3 from apache/master

update

commit 0042344cfb67900a1c815da932e5ea4c7ba93a19
Author: mashengchen 
Date:   2015-12-30T10:32:40Z

Merge pull request #4 from apache/master

update head

commit 11f9686b5df87f70b79edac5621da0610c7e70b8
Author: mashengchen 
Date:   2016-01-11T02:49:44Z

jira1729

commit cac96bc7ea69a3f1293e9ddef5e752f8785fb886
Author: mashengchen 
Date:   2016-01-11T03:19:05Z

Merge pull request #5 from apache/master

update head

commit 62237083f0db8230e257b6a8a12b5ca21c3831d8
Author: mashengchen 
Date:   2016-01-11T04:03:24Z

Merge branch 'master' of github.com:apache/incubator-trafodion

commit 5292fdc470db2317f4ab1133d298e98b97992e1e
Author: mashengchen 
Date:   2016-01-11T04:19:31Z

Merge branch 'master' of github.com:mashengchen/incubator-trafodion

commit 0ea8552120bd12d8c5ed1757325276cca3eac172
Author: mashengchen 
Date:   2016-01-11T05:39:43Z

Update HBaseClient.java

remove unnecessary log.

commit 6e0193a1837148fe5fecd3653402ddaff61f54ce
Author: mashengchen 
Date:   2016-01-11T05:41:00Z

Update CoprocessorUtils.java

change log level

commit 0dbaf3b3164069587a854968869211d1b69c3434
Author: mashengchen 
Date:   2016-01-11T05:52:45Z

jira1729




> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[liu ming (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15087053#comment-15087053
 ] 

liu ming commented on TRAFODION-1729:
-

Hi, Stack,
As per my understanding, some configuration items are client-side . (I don't 
know if there is such a concept in HBase .. ). For example: hbase.rpc.timeout
I think a client app can write code like this :
  Configuration conf = HBaseConfiguration.create();
  conf.set(“hbase.rpc.timeout”,”6”);
to change this setting , without HBase master or Region Server to restart for 
that new timeout to take effect.

But settings like 'hbase.snapshot.master.timeoutMillis' is a server-side 
config, and need a restart.

In fact, this is one of the confusing concept for me. Both client and HBase 
server use hbase-site.xml , and I don't find an article talking about 
differences. Some can be changed in app side, some not.

BTW, I am studying how to submit a patch to HBase, it will take some time. And 
I file a JIRA already:
https://issues.apache.org/jira/browse/HBASE-15076
And get a comment, it seems I need to study HBase internals more, it is not a 
single line of change.

thanks,
Ming

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[David Wayne Birdsall (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085810#comment-15085810
 ] 

David Wayne Birdsall commented on TRAFODION-1729:
-

Hi Ming,

So today Trafodion can offer transactional support on native HBase tables. Your 
last comment has me wondering how this will work in your new proposal. I am 
guessing that transactional support would only be available on native HBase 
tables that are opened with the transactional coprocessors?

Dave

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[liu ming (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086429#comment-15086429
 ] 

liu ming commented on TRAFODION-1729:
-

Thanks Sean,

After this change, and 1730, there are still these items seems not able to 
avoid HBase restart:
hbase.snapshot.master.timeoutMillis 60
hbase_regionserver_lease_period 60
hbase.regionserver.region.split.policy 
org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
hbase.snapshot.enabled true
hbase.master.distributed.log.splitting false

My hope is :
hbase.snapsot.enable is by default true, and in most case it is. So we don't 
need to change it.
I think with new improvement, we can allow distributed log splitting in HBase. 
In fact, I am not quite sure if Trafodion real change this or not, and why we 
need this. 
For others, I still need to investigate, but seems they are no longer 'forced' 
changes (some timeouts), so we can ask user to change it in Installer, but not 
a forced step, an optional step.

thanks,
Ming

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[liu ming (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086433#comment-15086433
 ] 

liu ming commented on TRAFODION-1729:
-

Hi, Dave,

As per my understanding, Trafodion cannot support ACID on native HBase tables 
access. If this is a requirement, yes, this is a problem.
However, I feel it is still possible, we can alter the HBase table to add 
coprocessor when Trafodion need to transactionally access it. I will do an 
prototype. 
But I think we first need to confirm: Is now Trafodion support ACID on native 
HBase tables?

thanks,
Ming

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[liu ming (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086565#comment-15086565
 ] 

liu ming commented on TRAFODION-1729:
-

Thanks Narendra,

Then, this may be a very big issue. I will try to do some testing. The solution 
in my mind is before access the native HBase table, call method to alter the 
target table, to add trafodion coprocessors at runtime, when close the table, 
remove trafodion coprocessors.
But I don't know if HBase support this or not, need some investigation.

Ming

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[Narendra Goyal (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086477#comment-15086477
 ] 

Narendra Goyal commented on TRAFODION-1729:
---

Hi Ming,

Yes, we do support ACID on native HBase tables.

Thanks,
-Narendra

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  
> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
> hbase.snapshot.enabled true
> hbase.bulkload.staging.dir/hbase-staging
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086909#comment-15086909
 ] 

stack commented on TRAFODION-1729:
--

bq  there are still these items seems not able to avoid HBase restart:

What does this mean? You want to set configuration but it requires a restart? 
Most all changes in hbase configuration require restart, not?

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  
> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[Sean Broeder (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086287#comment-15086287
 ] 

Sean Broeder commented on TRAFODION-1729:
-

Ming's proposal sounds good to me as it could potentially allow for different 
coprocessors to be loaded for different tables.  We have 2 coprocessors in 
development, so this may allow them to work in harmony.

Also, hbase.regionserver.region.transactional.tlog is no longer required.

Sean

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[liu ming (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084500#comment-15084500
 ] 

liu ming commented on TRAFODION-1729:
-

Thanks Stack, I need to learn how to submit a patch to HBase community.
And thanks Dave, here is the list of coprocessors:
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
in the future, there is one more:
org.apache.hadoop.hbase.coprocessor.transactional.SsccRegionEndpoint

The overhead of adding coprocessor is only involved when a Region is first 
open. And I assume there is no difference between we use addCoprocessor() java 
API to add, or we modify hbase-site.xml to add. Just two different ways to add 
coprocessor in HBase. hbase-site.xml is a global setting. In fact, it will let 
ALL hbase tables to load Trafodion coprocessors, even for those native hbase 
tables, which not created by Trafodion. With this change, we can control that 
only Trafodion tables will be equipped with these coprocessors. Yet another 
reason to do this change :-) 

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081339#comment-15081339
 ] 

stack commented on TRAFODION-1729:
--

[~limpo2000] Go ahead and file the issue against upstream and I'll commit it 
for you.

I'd be interested in how many CPs T8 loads too myself. Is there a list? Can we 
get rid of any of them?

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  
> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
> hbase.snapshot.enabled true

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[David Wayne Birdsall (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081309#comment-15081309
 ] 

David Wayne Birdsall commented on TRAFODION-1729:
-

This question is only tangentially related to this change, but I thought I'd 
ask anyway. How many co-processors does Trafodion use? If we add the 
co-processor(s) at Trafodion table create time, is it the same set that gets 
added each time? Is there any overhead to adding co-processors repeatedly? Is 
there any limit in HBase to the number of co-processors? Thanks.

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[liu ming (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080609#comment-15080609
 ] 

liu ming commented on TRAFODION-1729:
-

Hi, Stack,

Yes, the patch is exactly what Trafodion need, if that can be merged into HBase 
upstream, it will help T8 folks a lot! 
Thank you for considering this. So you will do that change in HBase?

thanks,
Ming

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
> tune the system to best suit Trafodion, but should not be a forced step. To 
> be note: Apache Phoenix is also a SQL on HBase, its installation will change 
> nothing of underlying HBase, very lightweight, no ‘intrude into’ the existing 
> HBase system. Trafodion is considered to be heavy and intrusive in this 
> manner, and I feel maybe we can change this.
>  
> Should I start this discussion in the dev mail list?
>  
> P.S. a list of changed config items. My proposal will remove the last one, 
> hope we can get rid of all of them:
> hbase.master.distributed.log.splitting  false
> hbase.snapshot.master.timeoutMillis  60
> hbase_regionserver_lease_period 60
> hbase.hregion.impl 
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> hbase.regionserver.region.split.policy  
> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076724#comment-15076724
 ] 

stack commented on TRAFODION-1729:
--

bq. Trafodion save the updated rows in RS heap, call it writeOrdering list, all 
Put and Delete are saved in this list and not go into HBase Region until 
commit, so other transaction cannot see the update. 

Might be better if we exposed the internal mvcc ... let a Coprocessor manage 
update thought that might be too much carnal knowledge. You'd need to be 
able to rollback... so yeah, probably best you keep the edits off to the side 
until time to commit.

bq. But that getScanner() method is protected, so Trafodion overload HRegion 
and make it public, in order to invoke it.

Make a patch for upstream HBase and we'll backport it all the ways to 1.0.x 
HBase. Something like this?

{code}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 0e9319c..d61589b 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -2579,7 +2579,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
 return scanner;
   }

-  protected RegionScanner getScanner(Scan scan, List 
additionalScanners,
+  public RegionScanner getScanner(Scan scan, List 
additionalScanners,
   boolean copyCellsFromSharedMem) throws IOException {
 startRegionOperation(Operation.SCAN);
 try {

{code}






> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[liu ming (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075963#comment-15075963
 ] 

liu ming commented on TRAFODION-1729:
-

thanks Stack very much for your comments. Let me try to clarify items one by 
one.
1. Overloading
To implement transaction semantics, Trafodion save the updated rows in RS heap, 
call it writeOrdering list, all Put and Delete are saved in this list and not 
go into HBase Region until commit, so other transaction cannot see the update. 
So, before commit, when it does a scan, Trafodion needs to filter out those 
'deleted' rows for current transaction, and plus those new put rows which still 
in memory not in the Region yet.
To archieve this, in the coprocessor, Trafodion needs to invoke a protected 
HRegion method:
protected RegionScanner getScanner(Scan scan,
   List additionalScanners,
   boolean copyCellsFromSharedMem)
throws IOException
To combine two scanners' result as final scan result. 
The first scanner will equiped with a filter to filter out 'deleted' rows which 
matching the delete objects in writeOrdering list. But new inserted rows or 
updated rows are Put objects in writeOrdering as well, so Trafodion need 
another additonalScanners to get those rows in the writeOrdering list and 
combine the result.
But that getScanner() method is protected, so Trafodion overload HRegion and 
make it public, in order to invoke it.
Currently, Trafodion overload the HRegion just to make that method as public, 
so can be invoked within the coprocessor.
Our proposal is to use Java's reflection technique to invoke this 'proctected' 
method without overload. It is used only once during a scanner construction, so 
we feel it will not impact the performance as well. This is the idea of this 
proposal.

2. Phoenix
I am wrong with Phoenix, I was playing with it long ago and as I remember, I 
just download a tar file and untar it and no change to my HBase system and get 
Phoenix working, so I said that, but probably my memeory is wrong, but the 
point is , we wish to avoid too much changes into HBase and restart it in order 
to use Trafodion.

And I don't know if it is possible to make that getScanner() method to public 
in future HBase release? If that can be made, it will help Trafodion 
development a lot.

thanks.

> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> 

[jira] [Commented] (TRAFODION-1729) change the coprocessor deployment method

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/TRAFODION-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075188#comment-15075188
 ] 

stack commented on TRAFODION-1729:
--

I like the sound of this issue.

Do we know what would need to change so you did not have to do this: {{To 
overload HRegion with TransactionalRegion}}  This is going to continue to cause 
you pain. Do you need it still now you have coprocessors (overloading was a 
technique that was done before we had coprocessors overloading is sure to 
break as hbase makes new releases).

bq.  I checked with Kevin and he proposed by using ‘java reflection’ we can 
also avoid this. 

Which method are you referring too? You want to return your own Scanner 
implementation?

bq. To be note: Apache Phoenix is also a SQL on HBase, its installation will 
change nothing of underlying HBase, very lightweight, no ‘intrude into’ the 
existing HBase system.

It is less intrusive, yes, but Phoenix requires CPs installed and has its 
fingers in hbase internals. We are working to get it out of hbase internals (if 
you need functionality of hbase that requires the overload of HRegoin or 
anything else, surface it upstream and we can change HBase or help you figure 
alternative).

Yeah, this belongs up on dev list, at least the general proposoal.  Good stuff.





> change the coprocessor deployment method
> 
>
> Key: TRAFODION-1729
> URL: https://issues.apache.org/jira/browse/TRAFODION-1729
> Project: Apache Trafodion
>  Issue Type: Improvement
>  Components: dtm
>Reporter: liu ming
>Assignee: mashengchen
>
> I have a proposal to change our current HBase coprocessor configuration 
> method. 
> There are three ways to add a coprocessor to a HBase table:
> 1.   Via editing hbase-site.xml, which will load coprocessor for ALL 
> tables (Trafodion is using this method now)
> 2.   Via HBase shell command
> 3.   Via HTableDescriptor.addCoprocessor() java API
> Trafodion now is using the first method. I proposed to use method 3, I 
> finished a prototype and test seems works well.
>  
> Here are the reasons I propose for this change:
> At present, the Trafodion installer needs to modify the hbase-site.xml and 
> then restart HBase instance for the configuration to take effect. This step 
> not only complicate the installer but also let user think Trafodion is 
> intrusive into underlying HBase system. It will be ideal if we can avoid this 
> step. Another problem: in CDH, there is a concept called ‘region server 
> group’ or something, so the settings will have to carefully handled by 
> installer to apply to all groups. As we saw recently in WebRoot deployment, 
> Trafodion failed due to this reason. All these are very error prone and 
> complicate the Trafodion installer. Once CDH or HDP changed something, 
> Trafodion may fail again.
>  
> So I spent time to investigate why we need to restart HBase in order to 
> install Trafodion.  
> As I understand, there are 3 major reasons 
> 1.   To add hbase-trx coprocessors
> 2.   To overload HRegion with TransactionalRegion
> 3.   Various configuration settings, need to check one by one.
> The first configuration can be avoided by applying my proposed change.
> The second one, I look through the TransactionalRegion.java, and find out the 
> only reason (now) is to overload the getScanner() method to be public so can 
> be invoked by the coprocessor. And there are only 1 or 2 places that API is 
> invoked in Trafodion code. I checked with Kevin and he proposed by using 
> ‘java reflection’ we can also avoid this. 
> All other configuration items to some extent look like ‘best to have’, but 
> not ‘must to have’. And I also find two config items seems never been used:
> hbase.bulkload.staging.dir /hbase-staging (Suresh can confirm, 
> but I search in all code, seems this is never used)
> hbase.regionserver.region.transactional.tlog   true (Narendra can 
> confirm, this is NEVER used, maybe a legacy config item?)
> Yes, by now, there are still some other config items seems cannot be avoided, 
> but I hope we can find some way to remove them in the future. I am not trying 
> to solve all issues right now, just want to start the effort to remove 
> unnecessary hbase reconfiguration.
> For this example, Coprocessors can be added to a table at run time, no need 
> to edit the hbase-site.xml and restart hbase. This is only the first step to 
> try to remove the deep impact to the current HBase config and restart HBase.
>  
> So I asked for your opinions about this change. If you think this is 
> necessary, I will continue to file a JIRA and fix it. 
>  
> I strongly recommend to get rid of the step of ‘modify hbase-site.xml and 
> restart your hbase’ for Trafodion installation, it should be an option , to 
>