Hello: I found an issue with HADOOP-10027 that :
SnappyCompressor can’t load native lib properly. Test TestHadoopnative is failing in hadoop-apache2 version 0.10 stacktrace: Caused by: java.lang.RuntimeException: native snappy library not available: SnappyCompressor has not been loaded. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:72) at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:195) at com.facebook.presto.hadoop.HadoopNative.loadAllCodecs(HadoopNative.java:71) at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:52) ... 24 more After look into the code found that in SnappyCompressor.java static { if (NativeCodeLoader.isNativeCodeLoaded() && NativeCodeLoader.buildSupportsSnappy()) { try { initIDs(); nativeSnappyLoaded = true; } catch (Throwable t) { LOG.error("failed to load SnappyCompressor", t); } } } the native method initIDs will complain: no such field clazz. > On Mar 21, 2017, at 4:32 PM, Poepping, Thomas <poe...@amazon.com> wrote: > > > > On 3/21/17, 3:17 PM, "Kuhu Shukla" <kshu...@yahoo-inc.com.INVALID> wrote: > > > +1 (non-binding) > > - Verified signatures. > - Downloaded and built from source tar.gz. > - Deployed a pseudo-distributed cluster on Mac Sierra. > - Ran example Sleep job successfully. > - Deployed latest Apache Tez 0.9 and ran sample Tez orderedwordcount > successfully. > > Thank you Junping and everyone else who worked on getting this release out. > > Warm Regards, > Kuhu > On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger > <ebad...@yahoo-inc.com.INVALID> wrote: > +1 (non-binding) > > - Verified checksums and signatures of all files > - Built from source on MacOS Sierra via JDK 1.8.0 u65 > - Deployed single-node cluster > - Successfully ran a few sample jobs > > Thanks, > > Eric > > On Tuesday, March 21, 2017 2:56 PM, John Zhuge <john.zh...@gmail.com> > wrote: > > > > +1. Thanks for the great effort, Junping! > > > - Verified checksums and signatures of the tarballs > - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3 > - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511 > - Cloud connectors: > - s3a: integration tests, basic fs commands > - adl: live unit tests, basic fs commands. See notes below. > - Deployed a pseudo cluster, passed the following sanity tests in > both insecure and SSL mode: > - HDFS: basic dfs, distcp, ACL commands > - KMS and HttpFS: basic tests > - MapReduce wordcount > - balancer start/stop > > > Needs the following JIRAs to pass all ADL tests: > > - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge. > - HDFS-11132. Allow AccessControlException in contract tests when > getFileStatus on subdirectory of existing files. Contributed by > Vishwajeet > Dusane > - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1 > runtime error. (John Zhuge via lei) > > > On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge <jzh...@cloudera.com> wrote: > >> Yes, it only affects ADL. There is a workaround of adding these 2 >> properties to core-site.xml: >> >> <property> >> <name>fs.adl.impl</name> >> <value>org.apache.hadoop.fs.adl.AdlFileSystem</value> >> </property> >> >> <property> >> <name>fs.AbstractFileSystem.adl.impl</name> >> <value>org.apache.hadoop.fs.adl.Adl</value> >> </property> >> >> I have the initial patch ready but hitting these live unit test failures: >> >> Failed tests: >> >> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest. >> testListStatus:257 >> expected:<1> but was:<10> >> >> Tests in error: >> >> TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest. >> testMkdirsFailsForSubdirectoryOfExistingFile:254 >> » AccessControl >> >> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest. >> testMkdirsFailsForSubdirectoryOfExistingFile:190 >> » AccessControl >> >> >> Stay tuned... >> >> John Zhuge >> Software Engineer, Cloudera >> >> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du <j...@hortonworks.com> wrote: >> >>> Thank you for reporting the issue, John! Does this issue only affect ADL >>> (Azure Data Lake) which is a new feature for 2.8 rather than other >> existing >>> FS? If so, I think we can leave the fix to 2.8.1 to fix given this is >> not a >>> regression and just a new feature get broken. >>> >>> >>> Thanks, >>> >>> >>> Junping >>> ------------------------------ >>> *From:* John Zhuge <jzh...@cloudera.com> >>> *Sent:* Monday, March 20, 2017 9:07 AM >>> *To:* Junping Du >>> *Cc:* common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; >>> yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org >>> *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3) >>> >>> Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No >>> FileSystem for scheme: adl". >>> >>> The issue were caused by backporting HADOOP-13037 to branch-2 and >> earlier. >>> HADOOP-12666 should not be backported, but some changes are needed: >>> property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml. >>> >>> I am working on a patch. >>> >>> >>> John Zhuge >>> Software Engineer, Cloudera >>> >>> On Fri, Mar 17, 2017 at 2:18 AM, Junping Du <j...@hortonworks.com> wrote: >>> >>>> Hi all, >>>> With fix of HDFS-11431 get in, I've created a new release candidate >>>> (RC3) for Apache Hadoop 2.8.0. >>>> >>>> This is the next minor release to follow up 2.7.0 which has been >>>> released for more than 1 year. It comprises 2,900+ fixes, improvements, >> and >>>> new features. Most of these commits are released for the first time in >>>> branch-2. >>>> >>>> More information about the 2.8.0 release plan can be found here: >>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release >>>> >>>> New RC is available at: http://home.apache.org/~junpin >>>> g_du/hadoop-2.8.0-RC3 >>>> >>>> The RC tag in git is: release-2.8.0-RC3, and the latest commit id >>>> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 >>>> >>>> The maven artifacts are available via repository.apache.org at: >>>> https://repository.apache.org/content/repositories/orgapachehadoop-1057 >>>> >>>> Please try the release and vote; the vote will run for the usual 5 >>>> days, ending on 03/22/2017 PDT time. > >>>> >>>> Thanks, >>>> >>>> Junping >>>> >>> >>> >> > > > > -- > John > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >