One correction: "... the oldest *active* branch." This would be 1.5.2-SNAPSHOT
I think some things, like the configuration generator, were only introduced in 1.6.0. Ultimately, you can choose what you'd like to target as long as the changes aren't invasive/breaking for the bugfix releases (1.5.2 and 1.6.1). On Jun 23, 2014 8:46 AM, "William Slacum" <[email protected]> wrote: > Work on the oldest branch possible and merge forward, please. > > > On Mon, Jun 23, 2014 at 6:00 AM, Hayden Marchant <[email protected]> > wrote: > > > Josh (and all who commented), > > > > Thanks for the comments. I'll take them into account, and will create the > > JIRAs. > > > > I was not intending on removing the CMS options, but rather only > including > > them in the JVM in which they are relevant, and including the equivalent > > in different JVMs (i.e. IBM ) - all through the bootstrap_config.sh. > > > > Here's my newbie question: Should I be making this patch based on 1.6.1, > > or should I always be working against the 'master' branch, and then > > backport the fix(es) to any desired older version? > > > > Regards, > > > > > > > > Hayden > > > > > > > > From: Josh Elser <[email protected]> > > To: [email protected], > > Date: 19/06/2014 06:43 PM > > Subject: Re: Running Accumulo on the IBM JVM > > > > > > > > <snip/> > > > > >>> b. > > >>> > > >>> > > >> > > > > > org.apache.accumulo.core.security.crypto.BlockedIOStreamTest.testGiantWrite. > > >>> This test assumes a max heap of about 1GB. This fails on IBM > > JRE, > > >>> since the default max heap is not specified, and on IBM JRE this > > depends > > >>> on the OS (see > > >>> > > >>> > > >> > > > > > http://www-01.ibm.com/support/knowledgecenter/SSYKE2_6.0.0/com.ibm.java.doc.diagnostics.60/diag/appendixes/defaults.html?lang=en > > > > >>> ). > > >>> Proposal: add -Xmx1g to the surefire maven plugin reference > > in > > >>> parent maven pom. > > >>> > > >> > > > This might be https://issues.apache.org/jira/browse/ACCUMULO-2774 > > > > Yup! I actually bumped this up to 1G already after I started seeing > > failures (again) from the ACCUMULO-2774 patch which set a 768M heap. > > Pull the upstream changes and feel free to submit something to address > > any problem you still have. > > > > > > > > > > >> > c. Both org.apache.accumulo.core.security.crypto.CrypoTest > > & > > >>> org.apache.accumulo.core.file.rfile.RFileTest have lots of failures > > due > > >> to > > >>> calls to SEcureRandom with Random Number Generator Provider > hard-coded > > as > > >>> Sun. The IBM JRE has it's own built in RNG Provider called IBMJCE. 2 > > >>> issues - hard-coded calls to SecureRandom.getInstance(<algo>,"SUN") > > and > > >>> also default value in Property class is "SUN". > > >>> Proposal: Add mechanism to override default Property through > > >>> System property through new annotator in Property class. Only usage > > will > > >>> be by Property.CRYPTO_SECURE_RNG_PROVIDER > > >> > > >> > > >> > > > I'm not sure about adding new annotators to Property. However, the > > > CryptoTest should be getting the value from the conf instead of > > hard-coding > > > it. Then you can specify the correct value in accumulo-site.xml > > > > > > I think another part of the issue is in > > > CryptoModuleFactory::fillParamsObjectFromStringMap because it looks > like > > > that ignores the default setting. > > > > > >> > > > >>> 2. Environment/Configuration > > >>> a. The generated configuration files contain references to > GC > > >>> params that are specific to Sun JVM. In accumulo-env.sh, the > > >>> ACCUMULO_TSERVER_OPTS contains -XX:NewSize and -XX:MaxNewSize , and > > also > > >>> in ACCUMULO_GENERAL_OPTS, > > >>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 are > > used. > > >>> b. in bin/accumulo, get ClassNotFoundException due to > > >>> specification of JAXP Doc Builder: > > >>> > > >>> > > >> > > > > > -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl > > >>> . > > >>> The Sun implementation of Document Builder Factory does not > > >> exists > > >>> in IBM JDK, so a ClassNotFoundException is thrown on running accumulo > > >>> script > > >>> > > >>> c. MiniAccumuloCluster - in the MiniAccumuloClusterImpl, > > >>> Sun-speciifc GC params are passed as params to the java process > > (similar > > >>> to section a. ) > > >>> > > >>> Single proposal for solving all three above issues: > > >>> Enhance bootstrap_config.sh with request to select Java > > vendor. > > >>> Selecting this will set correct values for GC params (they differ > > between > > >>> IBM and Sun), inclusion/ommision of JAXP setting. The > > >>> MiniAccumuloClusterImpl can read the same env variable that was set > in > > >>> code for the GC Params, and use in the exec command. > > >>> > > >> > > > I don't know enough about the IBM JDK to comment on this part > > > intelligently. Go ahead and generate a patch, and we can use that as a > > > starting point for discussion. > > > > I'm a little hesitant to remove the CMS configuration (as it really > > helps). My first thought about how to address this is you can submit > > some example Accumulo configurations that work with IBM JDK or you can > > add something to the configuration template/script (conf/examples and > > conf/templates with bin/bootstrap_config.sh, respectively). I think > > you're on the right path. > > > > > > > >> > > > >>> So far, my work has been focused on getting unit tests working for > > all > > >>> Java vendors in a clean manner. I have not yet run intensive testing > > of > > >>> real clusters following these changes, and would be happy to get > > pointers > > >>> to what else might need treatment. > > >> > > >> > > >> > > > Unit tests is a good first pass. Integration tests (mvn verify) is > > probably > > > the minimum that you want on your continuous integration once you have > > > things set up. > > > > > > Accumulo also comes with a set of longer running, cluster based tests, > > > since we know that there are some pieces too complex for unit tests to > > > catch. have a look in the test module for the Continuous Ingest test. > > Once > > > you get to that point, we can help you set it up if the README is > > unclear. > > > > > >> I would also like to hear if these changes make sense, and if so, > > should > > >>> I go ahead and create some JIRAs, and attach my patches for commit > > >>> approval? > > >>> > > >> > > > Filing JIRAs is going to be the most straightforward path, yes. > > > > > > > Looking forward to hearing feedback! > > > > Likewise. Looking forward to applying some patches! > > > > > > >
