[
https://issues.apache.org/jira/browse/HADOOP-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643042#comment-17643042
]
Steve Loughran commented on HADOOP-17563:
-----------------------------------------
ok, i'm seeing some quirks with spark master and hadoop trunk with the sbt
plugin missing bouncycastle.
{code}
[INFO] --- scala-maven-plugin:4.7.2:testCompile (scala-test-compile-first) @
spark-sql_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file:
/Users/stevel/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.7.1-bin_2.12.17__52.0-1.7.1_20220712T022208.jar
[INFO] compiler plugin:
BasicArtifact(com.github.ghik,silencer-plugin_2.12.17,1.7.10,null)
[INFO] compiling 603 Scala sources and 51 Java sources to
/Users/stevel/Projects/sparkwork/spark/sql/core/target/scala-2.12/test-classes
...
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/MyDoubleAvg.java:25:39:
[deprecation] UserDefinedAggregateFunction in
org.apache.spark.sql.expressions has been deprecated
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/MyDoubleSum.java:25:39:
[deprecation] UserDefinedAggregateFunction in
org.apache.spark.sql.expressions has been deprecated
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java:593:16:
[rawtypes] found raw type: Tuple2
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java:593:12:
[unchecked] unchecked call to Tuple2(T1,T2) as a member of the raw type Tuple2
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java:592:62:
[unchecked] unchecked method invocation: method asList in class Arrays is
applied to given types
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:89:48:
[rawtypes] found raw type: HashMap
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:89:44:
[unchecked] unchecked conversion
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/JavaHigherOrderFunctionsSuite.java:67:29:
[varargs] Varargs method could cause heap pollution from non-reifiable
varargs parameter objs
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/JavaHigherOrderFunctionsSuite.java:74:15:
[varargs] Varargs method could cause heap pollution from non-reifiable
varargs parameter ts
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/MyDoubleAvg.java:36:33:
[deprecation] UserDefinedAggregateFunction in
org.apache.spark.sql.expressions has been deprecated
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/MyDoubleSum.java:35:33:
[deprecation] UserDefinedAggregateFunction in
org.apache.spark.sql.expressions has been deprecated
[WARNING]
/Users/stevel/Projects/sparkwork/spark/sql/core/src/test/java/test/org/apache/spark/sql/connector/JavaAdvancedDataSourceV2WithV2Filter.java:72:10:
[rawtypes] found raw type: Literal
[ERROR] ## Exception when compiling 654 sources to
/Users/stevel/Projects/sparkwork/spark/sql/core/target/scala-2.12/test-classes
java.lang.NoClassDefFoundError:
org/bouncycastle/jce/provider/BouncyCastleProvider
java.lang.Class.getDeclaredMethods0(Native Method)
java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
java.lang.Class.privateGetPublicMethods(Class.java:2902)
java.lang.Class.getMethods(Class.java:1615)
sbt.internal.inc.ClassToAPI$.toDefinitions0(ClassToAPI.scala:173)
sbt.internal.inc.ClassToAPI$.$anonfun$toDefinitions$1(ClassToAPI.scala:125)
scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:454)
sbt.internal.inc.ClassToAPI$.toDefinitions(ClassToAPI.scala:125)
sbt.internal.inc.ClassToAPI$.$anonfun$process$1(ClassToAPI.scala:34)
scala.collection.immutable.List.foreach(List.scala:333)
sbt.internal.inc.ClassToAPI$.process(ClassToAPI.scala:34)
sbt.internal.inc.javac.AnalyzingJavaCompiler.readAPI$1(AnalyzingJavaCompiler.scala:185)
sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$22(AnalyzingJavaCompiler.scala:215)
sbt.internal.inc.classfile.JavaAnalyze$.readInheritanceDependencies$1(JavaAnalyze.scala:186)
sbt.internal.inc.classfile.JavaAnalyze$.$anonfun$apply$16(JavaAnalyze.scala:193)
sbt.internal.inc.classfile.JavaAnalyze$.$anonfun$apply$16$adapted(JavaAnalyze.scala:108)
scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
scala.collection.AbstractIterable.foreach(Iterable.scala:926)
scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:896)
sbt.internal.inc.classfile.JavaAnalyze$.apply(JavaAnalyze.scala:108)
sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$21(AnalyzingJavaCompiler.scala:215)
sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$21$adapted(AnalyzingJavaCompiler.scala:207)
scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
scala.collection.AbstractIterable.foreach(Iterable.scala:926)
scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:896)
sbt.internal.inc.javac.AnalyzingJavaCompiler.$anonfun$compile$19(AnalyzingJavaCompiler.scala:207)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
sbt.internal.inc.javac.AnalyzingJavaCompiler.timed(AnalyzingJavaCompiler.scala:262)
sbt.internal.inc.javac.AnalyzingJavaCompiler.compile(AnalyzingJavaCompiler.scala:207)
sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compileJava$2(MixedAnalyzingCompiler.scala:103)
sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compileJava$2$adapted(MixedAnalyzingCompiler.scala:91)
sbt.internal.inc.JarUtils$.withPreviousJar(JarUtils.scala:239)
sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compileJava$1(MixedAnalyzingCompiler.scala:91)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
sbt.internal.inc.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:248)
sbt.internal.inc.MixedAnalyzingCompiler.compileJava(MixedAnalyzingCompiler.scala:61)
sbt.internal.inc.MixedAnalyzingCompiler.compileJava0$1(MixedAnalyzingCompiler.scala:198)
sbt.internal.inc.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:211)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1$adapted(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.Incremental$.$anonfun$apply$5(Incremental.scala:179)
sbt.internal.inc.Incremental$.$anonfun$apply$5$adapted(Incremental.scala:177)
sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:463)
sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:116)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:263)
sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:418)
sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:506)
sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:405)
sbt.internal.inc.Incremental$.apply(Incremental.scala:171)
sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:181)
scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:364)
scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:122)
scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:89)
scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:310)
scala_maven.ScalaTestCompileMojo.execute(ScalaTestCompileMojo.java:51)
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
{code}
> Update Bouncy Castle to 1.68 or later
> -------------------------------------
>
> Key: HADOOP-17563
> URL: https://issues.apache.org/jira/browse/HADOOP-17563
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 3.3.1
> Reporter: Takanobu Asanuma
> Assignee: PJ Fanning
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.3.9
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> -Bouncy Castle 1.60 has Hash Collision Vulnerability. Let's update to 1.68.-
> Bouncy Castle 1.60 has the following vulnerabilities. Let's update to 1.68.
> * [https://nvd.nist.gov/vuln/detail/CVE-2020-26939]
> * [https://nvd.nist.gov/vuln/detail/CVE-2020-28052]
> * [https://nvd.nist.gov/vuln/detail/CVE-2020-15522]
> for anyone backporting this, note that recent bouncy castle jars are
> incompatible with older versions of asm.jar, and so older versions of spark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
