Vitali Obukh created HUDI-4735:
----------------------------------
Summary: Spark2 bundles made from master after 2022-07-23 failed
to stop
Key: HUDI-4735
URL: https://issues.apache.org/jira/browse/HUDI-4735
Project: Apache Hudi
Issue Type: Bug
Reporter: Vitali Obukh
Daily Spark2 Structured streaming application built against the latest change
on master started failing on 2022-07-23 with the error:
{quote}
22/07/28 10:52:50 ERROR org.apache.hudi.HoodieStreamingSink: Micro batch id=2
threw following exception:
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'test'
not found;
{quote}
Actually the database exists (created by the Hudi). Spark3 application with the
same code works fine.
Hudi options used for writing:
{code}
Map("hoodie.datasource.write.operation" -> "upsert",
"hoodie.datasource.write.table.type" -> "MERGE_ON_READ",
"hoodie.datasource.write.recordkey.field" -> s"$Meta.$Key",
"hoodie.datasource.write.precombine.field" -> s"$Meta.$Version",
"hoodie.datasource.write.payload.class" ->
"com.walmart.udps.hudi.integration.HudiTestAvroPayload",
"hoodie.compaction.payload.class" ->
"com.walmart.udps.hudi.integration.HudiTestAvroPayload",
"hoodie.datasource.write.partitionpath.field" -> TopLevelPartition,
"hoodie.datasource.write.hive_style_partitioning" -> "true",
"hoodie.datasource.hive_sync.enable" -> "true",
"hoodie.datasource.hive_sync.mode" -> "hms",
"hoodie.datasource.hive_sync.database" -> dbName(),
"hoodie.datasource.hive_sync.partition_fields" -> TopLevelPartition,
"hoodie.datasource.compaction.async.enable" -> "false"
)
{code}
Cloud platform - Google Dataproc.
Spark version - 3.1.3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
