Hi, all, Very glad we are discussing this. Although I think this is more suitable for the dev mail list ☺
First reply to Amanda’s question , “hbase-site hbase.regionserver.region.transactional.tlog true “ is no longer need, you can remove it from traf_*modes for sure. I also strongly think “hbase-site hbase.bulkload.staging.dir /hbase-staging” is no longer required, but I cannot make 100% sure of it. Bulkloader expert should confirm it. The only reference seems like a dead code for me, HBulkLoadClient.java line 96. And, I have started a similar discussion for some other traf_*mods changes at Jira 1729/1730.. With TRAFODION-1729, there is no need for the Trafodion installer to change this specific hbase-site.xml setting : “hbase-site hbase.coprocessor.region.classes”. But we leave it there in installer, since there are still needs to have hbase-site.xml changed for some developers to easily change them for debug purpose. But I really think no end user ever need to understand it or even config it. And, there are still some serious concerns about this JIRA, so we keep current behavior unchanged until all people are confident about this change. ☺ ( In fact, there is already a problem reported and solved by Dave due to this checkin…, but it is a minor issue comparing the problem it aims to solve), But technically, this item can now be removed from traf_*mod change list, Trafodion is smart enough now to add required coprocessor on the fly , if hbase-site.xml doesn’t have these settings. With TRAFODION-1730, which in fact solved by HBase community (HBASE-15076), it is also possible the installer no longer need to change the “hbase-site hbase.hregion.impl” settings soon. However, That HBase change is merged into HBase R1.2, so until Trafodion can support HBase R1.2, installer still need to change this setting item. And Trafodion DTM code need some change for this as well, which not done yet. For Gunnar’s documentation purpose, i.e. why Trafodion needs to change this item? I have a lengthy explanation at HBASE-15076, don’t want to paste that long text here. https://issues.apache.org/jira/browse/HBASE-15076?filter=-2 So sorry that although we are working on these two jiras, installer must still take care of them for now… And as Amanda listed, there are still more items unknown than known, so I really hope more people can join this discussion and we can finally remove all of them if possible, at least minimize the list. Although a long reply, the only item I came out for sure now is one can remove the “hbase.regionserver.region.transactional.tlog” from traf_*mod scripts... Thanks, Ming 发件人: Amanda Moran [mailto:[email protected]] 发送时间: 2016年2月10日 1:43 收件人: [email protected] 主题: Re: traf_*mods in trafodion installation Amanda: it should be possible and desireable to make all these configuration changes in advance. They're totally valid HBase configuration settings that could very-well exist in advance. [Amanda]: I would like to know the smallest number of these that must be set in order for Trafodion to be installer. Are all these settings mandatory for installation/start? This is what I want/need to know. Does a user have to go and set all those settings by hand in advance of installation Trafodion (assuming we 'remove' the traf_*mods scripts)? On Tue, Feb 9, 2016 at 9:32 AM, Gunnar Tapper <[email protected]<mailto:[email protected]>> wrote: Amanda: it should be possible and desireable to make all these configuration changes in advance. They're totally valid HBase configuration settings that could very-well exist in advance. The worse situation is if the Trafodion settings clash with some setting that the customer already has for some other reason. For example, can we really change the staging setting without affecting something else? This is why I am documenting these settings as requirements in the upcoming Provisioning Guide with the reasoning behind the setting. Further, I really recommend that we separate Trafodion installation from configuration changes in the infrastructure to ensure that it's possible to change the configuration changes, test their impact, and THEN install Trafodion. The testing could consist of something as simple as verify the restart worked or as advanced as an application regression test. Very user specific and dependent what else HDFS and HBase is being used for. For example, what's the impact on the rest of the Hadoop environment if turning on HDFS ACLs? Thanks, Gunnar Gunnar On Tue, Feb 9, 2016 at 9:45 AM, Amanda Moran <[email protected]<mailto:[email protected]>> wrote: I agree with that :) Thanks Qifan! On Tue, Feb 9, 2016 at 8:43 AM, Qifan Chen <[email protected]<mailto:[email protected]>> wrote: Hi Amanda, I know the topic that I brought up is a little bit different :-). But the objectives of both thoughts are pretty much in line with the improvement of install experiences. Thank you. --Qifan On Tue, Feb 9, 2016 at 10:36 AM, Amanda Moran <[email protected]<mailto:[email protected]>> wrote: Thanks for the response Qifan... but I am not sure we are talking about the same thing. I am talking about the traf_<hadoop distro>_mods98 script that is run as apart of the Trafodion Installation scripts. It is used to setup HBase settings, copy hbase-trx*.jar file, and restart HBase, HDFS, and Zookeeper. I would like to know what are the smallest number of settings that need to be set in advance for Trafodion to install/start. Examples of things set: hbase-site hbase.master.distributed.log.splitting false hbase-site hbase.coprocessor.region.classes "org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,org.apache.hadoop.hbase.coprocessor.AggregateImplementation" hbase-site hbase.hregion.impl org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion hbase-site hbase.regionserver.region.split.policy org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy hbase-site hbase.snapshot.enabled true hbase-site hbase.bulkload.staging.dir /hbase-staging hbase-site hbase.regionserver.region.transactional.tlog true hbase-site hbase.snapshot.master.timeoutMillis 600000 hbase-site hbase.snapshot.region.timeout 600000 hdfs-site dfs.namenode.acls.enabled true hbase-site hbase.client.scanner.timeout.period 600000 Thanks all! On Thu, Feb 4, 2016 at 5:40 AM, Qifan Chen <[email protected]<mailto:[email protected]>> wrote: Hi Amanda, I like your idea. On Trafodion's modification of config files for HBase, HDFS, and Zookeeper, I wonder if it is possible to re-difine the port range for an install, without going through all the install steps. This has the practical value that due to port conflict, an install could not start successfully, and one can easily switch to a different port range to retry with this capability. In addition, some of the install steps, such as TPCDS data set generation, and conversion to UTF8, could be made parallel to shorten the install duration. They are done one table at a time in serial mode now. TPCDS table data is generated through the command "dsdgen -scale ..." which could take the option "-parallel". Data conversion to UTF8 is done through the iconv utility. Regards, --Qifan On Wed, Feb 3, 2016 at 10:34 PM, Amanda Moran <[email protected]<mailto:[email protected]>> wrote: HI there All- The traf_*_mods scripts are used during installation to set HBase, HDFS, and Zookeeper settings needed by Trafodion. The script takes a long time to run (slowing down installation by a lot) because it needs to restart all these services for the changes to be picked up. My question is: What if I wanted to set the settings in advance of running the installer. What would be the absolute minimum set of settings that I need to get installed? I could set other settings later with a script or by hand. Thanks! -- Thanks, Amanda Moran -- Regards, --Qifan -- Thanks, Amanda Moran -- Regards, --Qifan -- Thanks, Amanda Moran -- Thanks, Gunnar If you think you can you can, if you think you can't you're right. -- Thanks, Amanda Moran
