@Attila, would you like me to open jira for: - HADOOP-13444 replaced org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets so it shall not be a problem on hadoop 2.8+
for 4.3.1 it's probably a non-issue but for 5.0 perhaps we'd like to tackle it? I'm happy to dedicate some cycles to it. On Tue, Jan 30, 2018 at 11:52 PM, Attila Sasvari <asasv...@cloudera.com> wrote: > - I could not reproduce the NoClassDefFoundError Artem reported with hadoop > 2.6.0. I changed bin/oozie-setup.sh so that it prints loaded classes > (passed -verbose:class to the JVM). Interestingly, org/apache/commons/io/ > Charsets was not even loaded. > > However, when I tried again with hadoop.version *2.7.5* , it resulted in > the error mentioned: > > java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets > at org.apache.hadoop.security.Credentials.<clinit>(Credentials.java:222) > at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334) > at org.apache.oozie.service.HadoopAccessorService.createJobConf( > HadoopAccessorService.java:349) > at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:171) > at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:67) > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.io.Charsets > > It looks like https://issues.apache.org/jira/browse/OOZIE-2396 > > Some more info: > - ./lib/commons-io-2.1.jar does not contain org/apache/commons/io/ > Charsets. > > - related hadoop 2.7.0 jira: > https://issues.apache.org/jira/browse/HADOOP-11389 > - related hadoop code: > https://github.com/apache/hadoop/blame/branch-2.7.0/ > hadoop-common-project/hadoop-common/src/main/java/org/ > apache/hadoop/security/Credentials.java#L36 > - HADOOP-13444 replaced org.apache.commons.io.Charsets with > java.nio.charset.StandardCharsets so it shall not be a problem on hadoop > 2.8+ > > Note: Oozie 5.0.0-beta1 includes OOZIE-2888 that add updated commons-io to > 2.4. That is why it is not a problem there. > > On Wed, Jan 31, 2018 at 1:18 AM, Satish Saley <satishsa...@apache.org> > wrote: > > > 1. However, json2.js is referenced by webapp/index.html and it > > is not found (not a blocker in my opinion). > > > > -> It will show only in firebug/developer tools. Not on UI. > > > > 2. [WARNING] Unable to process class com/sun/tools/jdi/ > ClassTypeImpl.class > > in > > JarAnalyzer File > > > > -> We can ignore the warning as build is going through > > > > 3. /user/oozie/share/lib/lib_20180130164518 > > Exception in thread "main" java.lang.NoClassDefFoundError: > > > > I am not able to reproduce this. > > > > saley@easyfamily-lm on Tue Jan 30 at 11:44 AM > > ~/src/oozie-4.3.1/distro/target/oozie-4.3.1-distro/oozie-4.3.1 > > $ bin/oozie-setup.sh sharelib create -fs hdfs://localhost -concurrency 8 > > -locallib oozie-sharelib-4.3.1.tar.gz > > setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" > > log4j:WARN No appenders could be found for logger > > (org.apache.hadoop.util.Shell). > > log4j:WARN Please initialize the log4j system properly. > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > > more info. > > SLF4J: Class path contains multiple SLF4J bindings. > > SLF4J: Found binding in > > [jar:file:/Users/saley/src/oozie-4.3.1/distro/target/ > > oozie-4.3.1-distro/oozie-4.3.1/lib/slf4j-log4j12-1.6.6.jar! > > /org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: Found binding in > > [jar:file:/Users/saley/src/oozie-4.3.1/distro/target/ > > oozie-4.3.1-distro/oozie-4.3.1/lib/slf4j-simple-1.6.6.jar!/ > org/slf4j/impl/ > > StaticLoggerBinder.class] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > explanation. > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > the destination path for sharelib is: > > /user/saley/share/lib/lib_20180130114509 > > Running 410 copy tasks on 8 threads > > > > > > saley@easyfamily-lm on Tue Jan 30 at 11:45 AM > > ~/src/oozie-4.3.1/distro/target/oozie-4.3.1-distro/oozie-4.3.1 > > $ hdfs dfs -ls > > SLF4J: Class path contains multiple SLF4J bindings. > > SLF4J: Found binding in > > [jar:file:/Users/saley/hadoop-2.6.3/share/hadoop/common/lib/ > > slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: Found binding in > > [jar:file:/Users/saley/src/tez/tez-dist/target/tez-0.8.5- > > minimal/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/ > > StaticLoggerBinder.class] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > explanation. > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > 18/01/30 11:45:27 WARN util.NativeCodeLoader: Unable to load > native-hadoop > > library for your platform... using builtin-java classes where applicable > > Found 1 items > > drwxr-xr-x - saley supergroup 0 2018-01-30 11:45 share > > > > > > > > > > On Tue, Jan 30, 2018 at 8:52 AM, Artem Ervits <artemerv...@gmail.com> > > wrote: > > > > > yes I realized that after hitting send, I encountered the following > > issues > > > but it may just be me > > > > > > hitting https://issues.apache.org/jira/browse/BCEL-173 when compiling > > with > > > > > > bin/mkdistro.sh -Phadoop-2 -DskipTests -Puber -DtargetJavaVersion=1.8 > > > -DjavaVersion=1.8 -Dhadoop.version=2.7.5 > > > > > > [WARNING] Unable to process class com/sun/tools/jdi/ > ClassTypeImpl.class > > in > > > JarAnalyzer File > > > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4. > > > x86_64/jre/../lib/tools.jar > > > org.apache.bcel.classfile.ClassFormatException: Invalid byte tag in > > > constant pool: 18 > > > at org.apache.bcel.classfile.Constant.readConstant(Constant.java:146) > > > at org.apache.bcel.classfile.ConstantPool.<init>(ConstantPool.java:67) > > > at > > > org.apache.bcel.classfile.ClassParser.readConstantPool( > > > ClassParser.java:222) > > > at org.apache.bcel.classfile.ClassParser.parse(ClassParser.java:136) > > > at > > > org.apache.maven.shared.jar.classes.JarClassesAnalysis. > > > analyze(JarClassesAnalysis.java:92) > > > at > > > org.apache.maven.report.projectinfo.dependencies.Dependencies. > > > getJarDependencyDetails(Dependencies.java:276) > > > at > > > org.apache.maven.report.projectinfo.dependencies. > > > renderer.DependenciesRenderer.hasSealed(DependenciesRenderer.java: > 1374) > > > at > > > org.apache.maven.report.projectinfo.dependencies. > > > renderer.DependenciesRenderer.renderSectionDependencyFileDet > > > ails(DependenciesRenderer.java:544) > > > at > > > org.apache.maven.report.projectinfo.dependencies. > > > renderer.DependenciesRenderer.renderBody( > DependenciesRenderer.java:271) > > > at > > > org.apache.maven.reporting.AbstractMavenReportRenderer.render( > > > AbstractMavenReportRenderer.java:79) > > > at > > > org.apache.maven.report.projectinfo.DependenciesReport.executeReport( > > > DependenciesReport.java:206) > > > at > > > org.apache.maven.reporting.AbstractMavenReport.generate( > > > AbstractMavenReport.java:190) > > > at > > > org.apache.maven.report.projectinfo.AbstractProjectInfoReport.execute( > > > AbstractProjectInfoReport.java:202) > > > at > > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( > > > DefaultBuildPluginManager.java:101) > > > at > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > > > MojoExecutor.java:209) > > > at > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > > > MojoExecutor.java:153) > > > at > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > > > MojoExecutor.java:145) > > > at > > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder. > buildProject( > > > LifecycleModuleBuilder.java:84) > > > at > > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder. > buildProject( > > > LifecycleModuleBuilder.java:59) > > > at > > > org.apache.maven.lifecycle.internal.LifecycleStarter. > > singleThreadedBuild( > > > LifecycleStarter.java:183) > > > at > > > org.apache.maven.lifecycle.internal.LifecycleStarter. > > > execute(LifecycleStarter.java:161) > > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) > > > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) > > > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) > > > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) > > > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java: > > > 62) > > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > org.codehaus.plexus.classworlds.launcher.Launcher. > > > launchEnhanced(Launcher.java:290) > > > at > > > org.codehaus.plexus.classworlds.launcher.Launcher. > > > launch(Launcher.java:230) > > > at > > > org.codehaus.plexus.classworlds.launcher.Launcher. > > > mainWithExitCode(Launcher.java:414) > > > at org.codehaus.plexus.classworlds.launcher.Launcher. > > > main(Launcher.java:357) > > > > > > Then after a successful build getting the following version string > > > > > > Oozie distro created, DATE[2018.01.30-16:22:57GMT] VC-REV[unavailable], > > > available at [/tmp/oozie-4.3.1/distro/target] > > > > > > we can now pull VC-REV with https://issues.apache.org/ > > > jira/browse/OOZIE-3152 > > > > > > finally saw the following > > > > > > [oozie@hadoop oozie-4.3.1]$ bin/oozie-setup.sh sharelib create -fs > > > hdfs://localhost:9000 -concurrency 8 -locallib > > oozie-sharelib-4.3.1.tar.gz > > > setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" > > > log4j:WARN No appenders could be found for logger > > > (org.apache.hadoop.util.Shell). > > > log4j:WARN Please initialize the log4j system properly. > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig > for > > > more info. > > > SLF4J: Class path contains multiple SLF4J bindings. > > > SLF4J: Found binding in > > > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1- > > > distro/oozie-4.3.1/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/ > > > impl/StaticLoggerBinder.class] > > > SLF4J: Found binding in > > > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1- > > > distro/oozie-4.3.1/lib/slf4j-simple-1.6.6.jar!/org/slf4j/ > > > impl/StaticLoggerBinder.class] > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > > explanation. > > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > > the destination path for sharelib is: > > > /user/oozie/share/lib/lib_20180130164518 > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > > org/apache/commons/io/Charsets > > > at org.apache.hadoop.security.Credentials.<clinit>( > Credentials.java:222) > > > at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334) > > > at > > > org.apache.oozie.service.HadoopAccessorService.createJobConf( > > > HadoopAccessorService.java:349) > > > at org.apache.oozie.tools.OozieSharelibCLI.run( > > OozieSharelibCLI.java:171) > > > at org.apache.oozie.tools.OozieSharelibCLI.main( > > OozieSharelibCLI.java:67) > > > Caused by: java.lang.ClassNotFoundException: > > > org.apache.commons.io.Charsets > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > ... 5 more > > > > > > none were blockers as I was able to get the following output > > > > > > $ curl http://localhost:11000/oozie/v2/admin/status > > > {"systemMode":"NORMAL"} > > > > > > > > > On Tue, Jan 30, 2018 at 10:45 AM, Rohini Palaniswamy < > roh...@apache.org> > > > wrote: > > > > > > > > the server directory is missing altogether now. > > > > That is as expected. It is supposed to be only in 5.0 > > > > > > > > On Mon, Jan 29, 2018 at 12:47 PM, Artem Ervits < > artemerv...@gmail.com> > > > > wrote: > > > > > > > > > the server directory is missing altogether now. > > > > > > > > > > On Mon, Jan 29, 2018 at 6:59 AM, Attila Sasvari < > > asasv...@cloudera.com > > > > > > > > > wrote: > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > - Verified hash, signature files, rat report > > > > > > - Built Oozie distro with hadoop.version set to 2.6.0 using the > > uber, > > > > > > hadoop-2, spark-1 profiles > > > > > > - Submitted example workflows/coordinators/bundles (except hive > > > related > > > > > > ones) on a hadoop 2.6.0 in pseudo distributed mode > > > > > > - Verified Tomcat version was 6.0.53 > > > > > > - Clicked through the web UI, log retrieval worked, job DAG-s > were > > > > > > rendered, version was correct. > > > > > > - Verified instrumentation, SLA pages; they are now HTML based as > > > > > expected > > > > > > (OOZIE-2533). However, json2.js is referenced by > webapp/index.html > > > and > > > > it > > > > > > is not found (not a blocker in my opinion). > > > > > > - Enabled SLA, re-submitted sla coordinator example, and > retrieved > > > SLA > > > > > > details for the "one-op-wf" application, graph was displayed > > > properly. > > > > > > > > > > > > I used JDK 1.8.0_121 > > > > > > > > > > > > On Sun, Jan 28, 2018 at 7:09 AM, Satish Saley < > > > satishsa...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I have created a build for Oozie 4.3.1, candidate 3. > > > > > > > > > > > > > > It includes all the changes decided earlier except OOZIE-3018, > > > > > OOZIE-2771 > > > > > > > and > > > > > > > OOZIE-3072. OOZIE-3018 makes changes to SchemaCheckerService > > which > > > we > > > > > are > > > > > > > not picking. > > > > > > > OOZIE-2771 makes changes to file created in OOZIE-2666. > > > > > > > OOZIE-3072 makes changes to files created in OOZIE-1770. > > > > > > > > > > > > > > > > > > > > > Keys to verify the signature of the release artifact are > > available > > > at > > > > > > > http://www.apache.org/dist/oozie/KEYS > > > > > > > > > > > > > > Please download, test, and try it out: > > > > > > > > > > > > > > http://people.apache.org/~satishsaley/oozie-4.3.1-rc3/ > > > > > > > > > > > > > > > > > > > > > The release, md5 signature, gpg signature, and rat report can > be > > > > found > > > > > at > > > > > > > the above address. > > > > > > > > > > > > > > Vote closes on 01/31/2018 11:59 PM PT. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Attila Sasvari > > > > > > Software Engineer > > > > > > <http://www.cloudera.com/> > > > > > > > > > > > > > > > > > > > > > > > > -- > -- > Attila Sasvari > Software Engineer > <http://www.cloudera.com/> >
