[
https://issues.apache.org/jira/browse/HUDI-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292615#comment-17292615
]
Volodymyr Burenin commented on HUDI-1063:
-----------------------------------------
Would be nice to see the used hadoop configuration and GCS connector versions.
I currently suspect that ' java.lang.NoSuchMethodError' has something to do
with mixed up dependencies, likely there are some incompatible dependencies
either in the spark image or somewhere else. One of the latest GCS connector is
incompatible with Hudi due to the dependencies.
> Save in Google Cloud Storage not working
> ----------------------------------------
>
> Key: HUDI-1063
> URL: https://issues.apache.org/jira/browse/HUDI-1063
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
> Reporter: David Lacalle Castillo
> Priority: Critical
> Labels: sev:critical, user-support-issues
> Fix For: 0.8.0
>
>
> I added to spark submit the following properties:
> {{--packages
> org.apache.hudi:hudi-spark-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4
> \ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'}}
> Spark version 2.4.5 and Hadoop version 3.2.1
>
> I am trying to save a Dataframe as follows in Google Cloud Storage as follows:
> tableName = "forecasts"
> basePath = "gs://hudi-datalake/" + tableName
> hudi_options = {
> 'hoodie.table.name': tableName,
> 'hoodie.datasource.write.recordkey.field': 'uuid',
> 'hoodie.datasource.write.partitionpath.field': 'partitionpath',
> 'hoodie.datasource.write.table.name': tableName,
> 'hoodie.datasource.write.operation': 'insert',
> 'hoodie.datasource.write.precombine.field': 'ts',
> 'hoodie.upsert.shuffle.parallelism': 2,
> 'hoodie.insert.shuffle.parallelism': 2
> }
> results = results.selectExpr(
> "ds as date",
> "store",
> "item",
> "y as sales",
> "yhat as sales_predicted",
> "yhat_upper as sales_predicted_upper",
> "yhat_lower as sales_predicted_lower",
> "training_date")
> results.write.format("hudi"). \
> options(**hudi_options). \
> mode("overwrite"). \
> save(basePath)
> I am getting the following error:
> Py4JJavaError: An error occurred while calling o312.save. :
> java.lang.NoSuchMethodError:
> org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V at
> io.javalin.core.util.JettyServerUtil.defaultSessionHandler(JettyServerUtil.kt:50)
> at io.javalin.Javalin.<init>(Javalin.java:94) at
> io.javalin.Javalin.create(Javalin.java:107) at
> org.apache.hudi.timeline.service.TimelineService.startService(TimelineService.java:102)
> at
> org.apache.hudi.client.embedded.EmbeddedTimelineService.startServer(EmbeddedTimelineService.java:74)
> at
> org.apache.hudi.client.AbstractHoodieClient.startEmbeddedServerView(AbstractHoodieClient.java:102)
> at
> org.apache.hudi.client.AbstractHoodieClient.<init>(AbstractHoodieClient.java:69)
> at
> org.apache.hudi.client.AbstractHoodieWriteClient.<init>(AbstractHoodieWriteClient.java:83)
> at
> org.apache.hudi.client.HoodieWriteClient.<init>(HoodieWriteClient.java:137)
> at
> org.apache.hudi.client.HoodieWriteClient.<init>(HoodieWriteClient.java:124)
> at
> org.apache.hudi.client.HoodieWriteClient.<init>(HoodieWriteClient.java:120)
> at
> org.apache.hudi.DataSourceUtils.createHoodieClient(DataSourceUtils.java:195)
> at
> org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:135)
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:108) at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
> at
> org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
> at
> org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676) at
> org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271) at
> org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at
> py4j.Gateway.invoke(Gateway.java:282) at
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at
> py4j.commands.CallCommand.execute(CallCommand.java:79) at
> py4j.GatewayConnection.run(GatewayConnection.java:238) at
> java.lang.Thread.run(Thread.java:748) (<class 'py4j.protocol.Py4JJavaError'>,
> Py4JJavaError('An error occurred while calling o312.save.\n', JavaObject
> id=o313), <traceback object at 0x7f33e56e6048>)
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
