lei-su-awx opened a new issue, #10484:
URL: https://github.com/apache/hudi/issues/10484
I used Spark 3.4.1 and hudi 0.14.0 on GKE, streaming reading a hudi COW
table(on GCS) and write to another hudi COW table(on GCS) with
upsert(RECORD_INDEX), here is my write option:
write_streaming_hudi_options = {
'hoodie.table.name': table_name,
'hoodie.datasource.write.recordkey.field': f'{primary_keys}, __region',
'hoodie.datasource.write.precombine.field': precombine_field,
'hoodie.datasource.write.table.name': table_name,
'hoodie.datasource.write.partitionpath.field': '__date',
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'hoodie.datasource.write.reconcile.schema': 'true',
'hoodie.schema.on.read.enable': 'true',
'hoodie.parquet.compression.codec': 'snappy',
'hoodie.datasource.write.hive_style_partitioning': 'true',
'hoodie.datasource.write.drop.partition.columns': 'true',
'hoodie.insert.shuffle.parallelism': '1000',
'hoodie.upsert.shuffle.parallelism': '1000',
'hoodie.datasource.write.payload.class':
'org.apache.hudi.common.model.DefaultHoodieRecordPayload',
'hoodie.metadata.record.index.enable': 'true',
'hoodie.index.type': 'RECORD_INDEX',
'hoodie.metadata.record.index.min.filegroup.count': '100',
'hoodie.metadata.optimized.log.blocks.scan.enable': 'true'
}
spark hudi jar is: `hudi-spark3-bundle_2.12-0.14.0.jar`
The steps that produces the exception are:
1. start the streaming pipeline with above option
2. write the first micro-batch to GCS successfully
3. second micro-batch throws an error
stack trace is:
<img width="2460" alt="image"
src="https://github.com/apache/hudi/assets/19327659/85b615c2-3c79-40e8-afbe-343175795fd2";>
So I have two question:
1. why this error happened?
2. why the first micro-batch always succeed?
**Environment Description**
* Hudi version : 0.14.0
* Spark version : 3.4.1
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : GCS
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
