bhasudha commented on issue #836:
URL: https://github.com/apache/incubator-hudi/issues/836#issuecomment-625529003
> just posting this in case anyone else hits this.
> i had hive (v 2.3.4) tables using
'com.uber.hoodie.hadoop.HoodieInputFormat' and COW hoodie files (written by
spark 2.3.x using hoodie v 0.4.6) working with prestodb 220, java8, there was a
custom hoodie jar in the presto plugin folder.
> i then tried to upgrade to prestosql 333 (which requires java11) but did
not change hive inputformat definitions/version or hoodie jar in presto or how
spark writing to the hoodie files. Simply trying presto (and JRE it runs on)
change...
>
> First hit this error on 333:
>
> ```
> Could not initialize class
com.uber.hoodie.common.table.timeline.HoodieActiveTimeline [DB
Errorcode=16777221]
> "failureInfo" : {
> "type" : "io.prestosql.spi.PrestoException",
> "message" : "Could not initialize class
com.uber.hoodie.common.table.timeline.HoodieActiveTimeline",
> "cause" : {
> "type" : "java.lang.NoClassDefFoundError",
> "message" : "Could not initialize class
com.uber.hoodie.common.table.timeline.HoodieActiveTimeline",
> "suppressed" : [ ],
> "stack" : [
"com.uber.hoodie.common.table.HoodieTableMetaClient.getActiveTimeline(HoodieTableMetaClient.java:192)",
"com.uber.hoodie.hadoop.HoodieInputFormat.listStatus(HoodieInputFormat.java:86)",
"org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)",
"io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:407)",
"io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:287)",
"io.prestosql.plugin.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:107)",
"io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:216)",
"io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)",
"io.prestosql.$gen.Presto_333____20200507_180703_2.run(Unknown Source)",
"io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)",
"java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)",
"java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)",
"java.base/java.lang.Thread.run(Unknown Source)" ],
> "errorCode" : {
> "code" : 65536,
> "name" : "GENERIC_INTERNAL_ERROR",
> "type" : "INTERNAL_ERROR"
> }
> },
> "suppressed" : [ ],
> "stack" : [
"io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:223)",
"io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)",
"io.prestosql.$gen.Presto_333____20200507_180703_2.run(Unknown Source)",
"io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)",
"java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)",
"java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)",
"java.base/java.lang.Thread.run(Unknown Source)" ],
> "errorCode" : {
> "code" : 16777221,
> "name" : "HIVE_UNKNOWN_ERROR",
> "type" : "EXTERNAL"
> }
> },
> ```
>
> then tried to replace the custom hoodie jar in presto plugin folder with
https://repo1.maven.org/maven2/com/uber/hoodie/hoodie-presto-bundle/0.4.7/hoodie-presto-bundle-0.4.7.jar
>
> hit this error:
>
> ```
> 2020-05-07T20:01:19.405Z ERROR main
io.prestosql.server.PrestoServer
'com.fasterxml.jackson.databind.ObjectMapper
com.fasterxml.jackson.databind.ObjectMapper.setDefaultPropertyInclusion(com.fasterxml.jackson.annotation.JsonInclude$Value)'
> java.lang.NoSuchMethodError: 'com.fasterxml.jackson.databind.ObjectMapper
com.fasterxml.jackson.databind.ObjectMapper.setDefaultPropertyInclusion(com.fasterxml.jackson.annotation.JsonInclude$Value)'
> at
io.airlift.json.ObjectMapperProvider.get(ObjectMapperProvider.java:128)
> at
io.airlift.json.ObjectMapperProvider.get(ObjectMapperProvider.java:47)
> at
com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85)
> at
com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77)
> at
com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:66)
> at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
> at
io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
> at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
> at
com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
> at
com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
> at
com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
> at
com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1094)
> at
io.airlift.json.JsonCodecFactory.createObjectMapper(JsonCodecFactory.java:130)
> at
io.airlift.json.JsonCodecFactory.jsonCodec(JsonCodecFactory.java:70)
> at io.airlift.json.JsonCodecProvider.get(JsonCodecProvider.java:43)
> at io.airlift.json.JsonCodecProvider.get(JsonCodecProvider.java:23)
> at
com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85)
> at
com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:57)
> at
com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:66)
> at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
> at
io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
> at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
> at
com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
> at
com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
> at
com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)
> at
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at
com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
> at
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
> at
com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
> at
com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
> at
com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
> at
com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
> at
com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
> at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
> at
io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
> at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
> at
com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
> at
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
> at
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
> at
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at
com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
> at
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
> at
com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
> at
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at
com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
> at
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
> at
com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
> at
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
> at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
> at com.google.inject.Guice.createInjector(Guice.java:87)
> at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:262)
> at
io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:118)
> at
io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:78)
> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at
io.prestosql.plugin.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:63)
> at
io.prestosql.connector.ConnectorManager.createConnector(ConnectorManager.java:349)
> at
io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:208)
> at
io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:200)
> at
io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:186)
> at
io.prestosql.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:88)
> at
io.prestosql.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:68)
> at io.prestosql.server.PrestoServer.run(PrestoServer.java:118)
> at io.prestosql.$gen.Presto_333____20200507_200106_1.run(Unknown
Source)
> at io.prestosql.server.PrestoServer.main(PrestoServer.java:72)
> ```
>
> then instead put
https://mvnrepository.com/artifact/org.apache.hudi/hudi-presto-bundle/0.5.2-incubating
in presto plugin folder and seems to work. So i guess new hudi bundle shades
jackson , caters for java11 reflection and can still read the uber inputformat
from hive definitions? @bhasudha
Thanks for updating this thread @tooptoop4 . I think the [pom changes](
https://github.com/apache/incubator-hudi/pull/846/files#diff-0aeecd78c8829509e3e17e8a87ddce2dR82)
and [java11 support](https://github.com/apache/incubator-hudi/pull/1369)
helped here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
