[
https://issues.apache.org/jira/browse/PARQUET-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796056#comment-16796056
]
ASF GitHub Bot commented on PARQUET-1544:
-----------------------------------------
gszadovszky commented on pull request #628: PARQUET-1544: Possible over-shading
of modules
URL: https://github.com/apache/parquet-mr/pull/628
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Possible over-shading of modules
> --------------------------------
>
> Key: PARQUET-1544
> URL: https://issues.apache.org/jira/browse/PARQUET-1544
> Project: Parquet
> Issue Type: Bug
> Reporter: Matt Cheah
> Assignee: Gabor Szadovszky
> Priority: Major
> Labels: pull-request-available
>
> When upgrading Apache Iceberg to use what's effectively the master branch of
> Parquet, we hit this stack trace in one of the unit tests:
>
> {{java.lang.NoSuchMethodError:
> org.apache.avro.Schema.setLogicalType(Lorg/apache/avro/LogicalType;)V at
> org.apache.avro.LogicalType.addToSchema(LogicalType.java:72) at
> com.netflix.iceberg.avro.TypeToSchema.<clinit>(TypeToSchema.java:42) at
> com.netflix.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:69) at
> com.netflix.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:65) at
> com.netflix.iceberg.PartitionData.getSchema(PartitionData.java:38) at
> com.netflix.iceberg.PartitionData.<init>(PartitionData.java:67) at
> com.netflix.iceberg.DataFiles.newPartitionData(DataFiles.java:41) at
> com.netflix.iceberg.DataFiles.access$000(DataFiles.java:36) at
> com.netflix.iceberg.DataFiles$Builder.<init>(DataFiles.java:225) at
> com.netflix.iceberg.DataFiles.builder(DataFiles.java:191) at
> com.netflix.iceberg.hive.HiveTablesTest.testConcurrentFastAppends(HiveTablesTest.java:108)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
> at
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
> at java.lang.Thread.run(Thread.java:748)}}
> At first it's not immediately obvious if this is a problem in Iceberg's
> dependency chain, or if it's a problem in how parquet-avro is packaged. But
> I'm filing here because it appears linked to PARQUET-1529, where we shaded
> FastUtil in various modules, including parquet-avro. A hypothesis is that
> applying the shading plugin caused references to Jackson 1 to be shaded as
> well, and that libraries that depend on both parquet-avro and Avro itself (as
> Iceberg does) are at risk of having mismatching implementations being loaded
> on the classpath.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
