Yeah, I'm right: we needed the 5711 hack in there ~ a million years ago to get the WordCountHBaseIT to work: https://www.google.com/search?q=crunch+hbase-5711
I wonder why it's back now. On Wed, Oct 15, 2014 at 5:33 AM, Josh Wills <jwi...@cloudera.com> wrote: > So applying the bit mask workaround from HBASE-5711 fixes the > HFileTargetIT test, but I have this deja vu-like feeling that we ran into > something like this before, but the Internet at this hotel isn't quite > solid enough for me to figure out when it was. Will keep poking around. > > J > > On Tue, Oct 14, 2014 at 9:17 PM, Josh Wills <jwi...@cloudera.com> wrote: > >> I can replicate the original error- looks like dfs.data.dir permissions >> aren't set properly in test mode for some reason. Will investigate and >> figure out a patch. >> >> >> On Tuesday, October 14, 2014, Micah Whitacre <mkw...@gmail.com> wrote: >> >>> One thing I noticed is that is when testing we don't force IPv4. >>> Zookeeper >>> does not handle IPv6 well and can actually significantly slow down tests >>> trying to resolve the localhost. This doesn't affect everyone because >>> your >>> /etc/hosts could help it resolve faster but does make machine setup >>> important. We could try adding an argline like this to surefire/failsafe >>> to enforce IPv4. >>> >>> <argLine>-Xmx512m -XX:MaxPermSize=128m >>> -Djava.net.preferIPv4Stack=true</argLine> >>> >>> On Tue, Oct 14, 2014 at 9:28 AM, Danushka Menikkumbura < >>> danushka.menikkumb...@gmail.com> wrote: >>> >>> > Thanks J! >>> > >>> > On Tue, Oct 14, 2014 at 9:57 AM, Josh Wills <jwi...@cloudera.com> >>> wrote: >>> > >>> > > I'm still rocking 1.7. Will give 1.8 a whirl this evening. >>> > > >>> > > J >>> > > >>> > > On Tue, Oct 14, 2014 at 6:53 AM, Danushka Menikkumbura < >>> > > danushka.menikkumb...@gmail.com> wrote: >>> > > >>> > > > I am on Ubuntu 14.04.1 LTS and Java 1.8.0_20. >>> > > > >>> > > > BTW maybe it is a result of the commit >>> > > > 3f98411364cec32a0a8c6681dfaabd43caa4dd60?. >>> > > > >>> > > > Thanks, >>> > > > Danushka >>> > > > >>> > > > >>> > > > >>> > > > On Tue, Oct 14, 2014 at 9:44 AM, Josh Wills <jwi...@cloudera.com> >>> > wrote: >>> > > > >>> > > > > No, master is compatible w/Hadoop 1; the error you're seeing is >>> > caused >>> > > by >>> > > > > the HBase testing code being flaky. I've experienced the flaky >>> > > > > HFileTargetIT test on my machine before, but never on a regular >>> > basis. >>> > > > Can >>> > > > > you give me the basics of your setup-- OS, java version, etc.? >>> > > > > >>> > > > > On Tue, Oct 14, 2014 at 6:38 AM, Danushka Menikkumbura < >>> > > > > danushka.menikkumb...@gmail.com> wrote: >>> > > > > >>> > > > > > No I don't. >>> > > > > > >>> > > > > > That means the master branch is not compatible with Hadoop 1? >>> > > > > > >>> > > > > > Thanks, >>> > > > > > Danushka >>> > > > > > >>> > > > > > On Tue, Oct 14, 2014 at 9:26 AM, Josh Wills < >>> jwi...@cloudera.com> >>> > > > wrote: >>> > > > > > >>> > > > > > > Hrm, okay. Do you get it if you use the -Dcrunch.platform=2 >>> > option? >>> > > > > > > >>> > > > > > > On Tue, Oct 14, 2014 at 6:23 AM, Danushka Menikkumbura < >>> > > > > > > danushka.menikkumb...@gmail.com> wrote: >>> > > > > > > >>> > > > > > > > Yes. I get it every time I try to build HBase module. >>> > > > > > > > >>> > > > > > > > FYI : org.apache.crunch.io.hbase.HFileTargetIT is the test >>> in >>> > > > error. >>> > > > > > > > >>> > > > > > > > Thanks, >>> > > > > > > > Danushka >>> > > > > > > > >>> > > > > > > > On Tue, Oct 14, 2014 at 9:15 AM, Josh Wills < >>> > jwi...@cloudera.com >>> > > > >>> > > > > > wrote: >>> > > > > > > > >>> > > > > > > > > That can happen intermittently if the local HBase cluster >>> > gets >>> > > > hung >>> > > > > > > up-- >>> > > > > > > > do >>> > > > > > > > > you get it regularly (i.e., every time you run?) >>> > > > > > > > > >>> > > > > > > > > J >>> > > > > > > > > >>> > > > > > > > > On Tue, Oct 14, 2014 at 6:08 AM, Danushka Menikkumbura < >>> > > > > > > > > danushka.menikkumb...@gmail.com> wrote: >>> > > > > > > > > >>> > > > > > > > > > Hi, >>> > > > > > > > > > >>> > > > > > > > > > I am getting the following test failure while building >>> > > Crunch. >>> > > > > Have >>> > > > > > > you >>> > > > > > > > > got >>> > > > > > > > > > an idea as to what may be the issue here?. >>> > > > > > > > > > >>> > > > > > > > > > 34787 [Thread-2057] INFO >>> > > > > > > > > > >>> > > > > > > > >>> > > > >>> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob >>> > > > > > - >>> > > > > > > > > > Job status available at: http://localhost:8080/ >>> > > > > > > > > > 44548 [M:0;danushka:38318.oldLogCleaner] ERROR >>> > > > > > > > > > org.apache.hadoop.hbase.client.HConnectionManager - >>> > > Connection >>> > > > > not >>> > > > > > > > found >>> > > > > > > > > > in the list, can't delete it (connection >>> > > > > > > > > > >>> > > > key=HConnectionKey{properties={hbase.zookeeper.quorum=localhost, >>> > > > > > > > > > hbase.rpc.timeout=60000, >>> > > > > hbase.zookeeper.property.clientPort=57963, >>> > > > > > > > > > zookeeper.znode.parent=/hbase, >>> > > hbase.client.retries.number=350, >>> > > > > > > > > > hbase.client.pause=100}, username='danushka'}). May be >>> the >>> > > key >>> > > > > was >>> > > > > > > > > > modified? >>> > > > > > > > > > java.lang.Exception >>> > > > > > > > > > at >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:493) >>> > > > > > > > > > at >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:429) >>> > > > > > > > > > at >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.stop(ReplicationLogCleaner.java:141) >>> > > > > > > > > > at >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> org.apache.hadoop.hbase.master.cleaner.CleanerChore.cleanup(CleanerChore.java:276) >>> > > > > > > > > > at org.apache.hadoop.hbase.Chore.run(Chore.java:94) >>> > > > > > > > > > at java.lang.Thread.run(Thread.java:745) >>> > > > > > > > > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time >>> > > elapsed: >>> > > > > > > 21.625 >>> > > > > > > > > sec >>> > > > > > > > > > >>> > > > > > > > > > Thanks, >>> > > > > > > > > > Danushka >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > -- >>> > > > > > > > > Director of Data Science >>> > > > > > > > > Cloudera <http://www.cloudera.com> >>> > > > > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills> >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > -- >>> > > > > > > Director of Data Science >>> > > > > > > Cloudera <http://www.cloudera.com> >>> > > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills> >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > Director of Data Science >>> > > > > Cloudera <http://www.cloudera.com> >>> > > > > Twitter: @josh_wills <http://twitter.com/josh_wills> >>> > > > > >>> > > > >>> > > >>> > > >>> > > >>> > > -- >>> > > Director of Data Science >>> > > Cloudera <http://www.cloudera.com> >>> > > Twitter: @josh_wills <http://twitter.com/josh_wills> >>> > > >>> > >>> >> >> >> -- >> Director of Data Science >> Cloudera <http://www.cloudera.com> >> Twitter: @josh_wills <http://twitter.com/josh_wills> >> >> > > > -- > Director of Data Science > Cloudera <http://www.cloudera.com> > Twitter: @josh_wills <http://twitter.com/josh_wills> > -- Director of Data Science Cloudera <http://www.cloudera.com> Twitter: @josh_wills <http://twitter.com/josh_wills>
