hudi-bot opened a new issue, #15398:
URL: https://github.com/apache/hudi/issues/15398
Daily Spark2 Structured streaming application built against the latest
change on master started failing on 2022-07-23 with the error:
{quote}
22/07/28 10:52:50 ERROR org.apache.hudi.HoodieStreamingSink: Micro batch
id=2 threw following exception:
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database
'test' not found;
{quote}
Actually the database exists (created by the Hudi). Spark3 application with
the same code works fine.
Hudi options used for writing:
{code}
Map("hoodie.datasource.write.operation" -> "upsert",
"hoodie.datasource.write.table.type" -> "MERGE_ON_READ",
"hoodie.datasource.write.recordkey.field" -> s"$Meta.$Key",
"hoodie.datasource.write.precombine.field" -> s"$Meta.$Version",
"hoodie.datasource.write.payload.class" ->
"com.walmart.udps.hudi.integration.HudiTestAvroPayload",
"hoodie.compaction.payload.class" ->
"com.walmart.udps.hudi.integration.HudiTestAvroPayload",
"hoodie.datasource.write.partitionpath.field" -> TopLevelPartition,
"hoodie.datasource.write.hive_style_partitioning" -> "true",
"hoodie.datasource.hive_sync.enable" -> "true",
"hoodie.datasource.hive_sync.mode" -> "hms",
"hoodie.datasource.hive_sync.database" -> dbName(),
"hoodie.datasource.hive_sync.partition_fields" -> TopLevelPartition,
"hoodie.datasource.compaction.async.enable" -> "false"
)
{code}
Cloud platform - Google Dataproc.
Spark version - 3.1.3.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4735
- Type: Bug
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
