Limess commented on issue #6367: URL: https://github.com/apache/hudi/issues/6367#issuecomment-1279833774
I finally got round to re-enabling the metadata table for this table - using Hudi 0.11.1 on EMR 6.8.0 now, and the same issue occurs: Here's the debug logs on an executor: ``` 22/10/15 20:50:06 INFO MapOutputTrackerWorker: Updating epoch to 24 and clearing cache 22/10/15 20:50:06 INFO TorrentBroadcast: Started reading broadcast variable 85 with 1 pieces (estimated total size 4.0 MiB) 22/10/15 20:50:06 INFO MemoryStore: Block broadcast_85_piece0 stored as bytes in memory (estimated size 219.1 KiB, free 4.6 GiB) 22/10/15 20:50:06 INFO TorrentBroadcast: Reading broadcast variable 85 took 2 ms 22/10/15 20:50:06 INFO MemoryStore: Block broadcast_85 stored as values in memory (estimated size 751.0 KiB, free 4.6 GiB) 22/10/15 20:50:06 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 23, fetching them 22/10/15 20:50:06 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:43817) 22/10/15 20:50:06 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 23, fetching them 22/10/15 20:50:06 INFO MapOutputTrackerWorker: Got the map output locations 22/10/15 20:50:06 INFO ShuffleBlockFetcherIterator: Getting 200 (111.3 KiB) non-empty blocks including 108 (59.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 92 (51.5 KiB) remote blocks 22/10/15 20:50:06 INFO ShuffleBlockFetcherIterator: Getting 200 (110.3 KiB) non-empty blocks including 108 (59.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 92 (50.9 KiB) remote blocks 22/10/15 20:50:06 INFO ShuffleBlockFetcherIterator: Started 2 remote fetches in 0 ms 22/10/15 20:50:06 INFO ShuffleBlockFetcherIterator: Started 2 remote fetches in 0 ms 22/10/15 20:50:06 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20221015204237661 for file bloom-filters-0001 22/10/15 20:50:06 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20221015204237661 for file bloom-filters-0002 22/10/15 20:50:06 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY 22/10/15 20:50:06 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY 22/10/15 20:50:06 INFO FileSystemViewManager: Creating in-memory based Table View 22/10/15 20:50:06 INFO FileSystemViewManager: Creating in-memory based Table View 22/10/15 20:50:06 INFO FileSystemViewManager: Creating InMemory based view for basePath s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata 22/10/15 20:50:06 INFO FileSystemViewManager: Creating InMemory based view for basePath s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata 22/10/15 20:50:06 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups 22/10/15 20:50:06 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups 22/10/15 20:50:06 INFO ClusteringUtils: Found 0 files in pending clustering operations 22/10/15 20:50:06 INFO AbstractTableFileSystemView: Building file system view for partition (bloom_filters) 22/10/15 20:50:06 INFO ClusteringUtils: Found 0 files in pending clustering operations 22/10/15 20:50:06 INFO AbstractTableFileSystemView: Building file system view for partition (bloom_filters) 22/10/15 20:50:06 DEBUG AbstractTableFileSystemView: #files found in partition (bloom_filters) =101, Time taken =41 22/10/15 20:50:06 DEBUG AbstractTableFileSystemView: #files found in partition (bloom_filters) =101, Time taken =42 22/10/15 20:50:06 DEBUG HoodieTableFileSystemView: Adding file-groups for partition :bloom_filters, #FileGroups=4 22/10/15 20:50:06 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=101, NumFileGroups=4, FileGroupsCreationTime=35, StoreTimeTaken=0 22/10/15 20:50:06 DEBUG AbstractTableFileSystemView: Time to load partition (bloom_filters) =76 22/10/15 20:50:06 DEBUG HoodieTableFileSystemView: Adding file-groups for partition :bloom_filters, #FileGroups=4 22/10/15 20:50:06 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=101, NumFileGroups=4, FileGroupsCreationTime=34, StoreTimeTaken=1 22/10/15 20:50:06 DEBUG AbstractTableFileSystemView: Time to load partition (bloom_filters) =77 22/10/15 20:50:06 DEBUG WriteMarkersFactory: Instantiated MarkerFiles with marker type: DIRECT 22/10/15 20:50:06 DEBUG WriteMarkersFactory: Instantiated MarkerFiles with marker type: DIRECT 22/10/15 20:50:06 INFO DirectWriteMarkers: Creating Marker Path=s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/bloom_filters/bloom-filters-0001_0-103-10538_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:07 INFO DirectWriteMarkers: Creating Marker Path=s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/bloom_filters/bloom-filters-0002_3-103-10541_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:07 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/bloom_filters/bloom-filters-0001_0-103-10538_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:07 INFO DirectWriteMarkers: [direct] Created marker file s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/bloom_filters/bloom-filters-0001_0-103-10538_20221015083833526.hfile.marker.APPEND in 319 ms 22/10/15 20:50:07 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer 22/10/15 20:50:07 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/bloom_filters/.bloom-filters-0001_20221015083833526.log.24_0-96-9648 22/10/15 20:50:07 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/bloom_filters/bloom-filters-0002_3-103-10541_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:07 INFO DirectWriteMarkers: [direct] Created marker file s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/bloom_filters/bloom-filters-0002_3-103-10541_20221015083833526.hfile.marker.APPEND in 359 ms 22/10/15 20:50:07 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer 22/10/15 20:50:07 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/bloom_filters/.bloom-filters-0002_20221015083833526.log.24_3-96-9651 22/10/15 20:50:07 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/bloom_filters/.bloom-filters-0001_20221015083833526.log.25_0-103-10538', fileLen=-1} 22/10/15 20:50:07 INFO HoodieAppendHandle: AppendHandle for partitionPath bloom_filters filePath bloom_filters/.bloom-filters-0001_20221015083833526.log.25_0-103-10538, took 588 ms. 22/10/15 20:50:07 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/bloom_filters/.bloom-filters-0001_20221015083833526.log.25_0-103-10538 22/10/15 20:50:07 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/bloom_filters/.bloom-filters-0002_20221015083833526.log.25_3-103-10541', fileLen=-1} 22/10/15 20:50:07 INFO MemoryStore: Block rdd_243_0 stored as values in memory (estimated size 590.0 B, free 4.6 GiB) 22/10/15 20:50:07 INFO Executor: Finished task 0.0 in stage 103.0 (TID 10538). 3006 bytes result sent to driver 22/10/15 20:50:07 INFO HoodieAppendHandle: AppendHandle for partitionPath bloom_filters filePath bloom_filters/.bloom-filters-0002_20221015083833526.log.25_3-103-10541, took 810 ms. 22/10/15 20:50:07 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/bloom_filters/.bloom-filters-0002_20221015083833526.log.25_3-103-10541 22/10/15 20:50:07 INFO MemoryStore: Block rdd_243_3 stored as values in memory (estimated size 592.0 B, free 4.6 GiB) 22/10/15 20:50:07 INFO Executor: Finished task 3.0 in stage 103.0 (TID 10541). 3006 bytes result sent to driver 22/10/15 20:50:13 INFO YarnCoarseGrainedExecutorBackend: Got assigned task 10548 22/10/15 20:50:13 INFO Executor: Running task 4.2 in stage 103.0 (TID 10548) 22/10/15 20:50:13 INFO ShuffleBlockFetcherIterator: Getting 10 (7.9 MiB) non-empty blocks including 3 (2.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 7 (5.5 MiB) remote blocks 22/10/15 20:50:13 INFO ShuffleBlockFetcherIterator: Started 2 remote fetches in 0 ms 22/10/15 20:50:14 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20221015204237661 for file col-stats-0001 22/10/15 20:50:14 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY 22/10/15 20:50:14 INFO FileSystemViewManager: Creating in-memory based Table View 22/10/15 20:50:14 INFO FileSystemViewManager: Creating InMemory based view for basePath s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata 22/10/15 20:50:14 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups 22/10/15 20:50:14 INFO ClusteringUtils: Found 0 files in pending clustering operations 22/10/15 20:50:14 INFO AbstractTableFileSystemView: Building file system view for partition (column_stats) 22/10/15 20:50:14 DEBUG AbstractTableFileSystemView: #files found in partition (column_stats) =25, Time taken =36 22/10/15 20:50:14 DEBUG HoodieTableFileSystemView: Adding file-groups for partition :column_stats, #FileGroups=2 22/10/15 20:50:14 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=25, NumFileGroups=2, FileGroupsCreationTime=6, StoreTimeTaken=0 22/10/15 20:50:14 DEBUG AbstractTableFileSystemView: Time to load partition (column_stats) =42 22/10/15 20:50:14 DEBUG WriteMarkersFactory: Instantiated MarkerFiles with marker type: DIRECT 22/10/15 20:50:14 INFO DirectWriteMarkers: Creating Marker Path=s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10548_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:14 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10548_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:14 INFO DirectWriteMarkers: [direct] Created marker file s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10548_20221015083833526.hfile.marker.APPEND in 126 ms 22/10/15 20:50:14 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer 22/10/15 20:50:14 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.11_1-0-1 22/10/15 20:50:16 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10548', fileLen=-1} 22/10/15 20:50:16 ERROR BaseSparkCommitActionExecutor: Error upserting bucketType UPDATE for partition :4 org.apache.hudi.exception.HoodieAppendException: Failed while appending records to s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10548 at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:410) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244) ~[__app__.jar:0.11.1-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:327) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.Task.run(Task.scala:138) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342] Caused by: java.lang.IllegalStateException: Writing multiple records with same key not supported for org.apache.hudi.common.table.log.block.HoodieHFileDataBlock at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.serializeRecords(HoodieHFileDataBlock.java:136) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:131) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:158) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:404) ~[__app__.jar:0.11.1-amzn-0] ... 31 more 22/10/15 20:50:16 WARN BlockManager: Putting block rdd_243_4 failed due to exception org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :4. 22/10/15 20:50:16 WARN BlockManager: Block rdd_243_4 could not be removed as it was not found on disk or in memory 22/10/15 20:50:16 ERROR Executor: Exception in task 4.2 in stage 103.0 (TID 10548) org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :4 at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:329) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244) ~[__app__.jar:0.11.1-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:327) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.Task.run(Task.scala:138) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342] Caused by: org.apache.hudi.exception.HoodieAppendException: Failed while appending records to s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10548 at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:410) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] ... 28 more Caused by: java.lang.IllegalStateException: Writing multiple records with same key not supported for org.apache.hudi.common.table.log.block.HoodieHFileDataBlock at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.serializeRecords(HoodieHFileDataBlock.java:136) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:131) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:158) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:404) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] ... 28 more 22/10/15 20:50:22 INFO YarnCoarseGrainedExecutorBackend: Got assigned task 10553 22/10/15 20:50:22 INFO Executor: Running task 4.5 in stage 103.0 (TID 10553) 22/10/15 20:50:22 INFO ShuffleBlockFetcherIterator: Getting 10 (7.9 MiB) non-empty blocks including 3 (2.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 7 (5.5 MiB) remote blocks 22/10/15 20:50:22 INFO ShuffleBlockFetcherIterator: Started 2 remote fetches in 0 ms 22/10/15 20:50:23 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20221015204237661 for file col-stats-0001 22/10/15 20:50:23 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY 22/10/15 20:50:23 INFO FileSystemViewManager: Creating in-memory based Table View 22/10/15 20:50:23 INFO FileSystemViewManager: Creating InMemory based view for basePath s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata 22/10/15 20:50:23 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups 22/10/15 20:50:23 INFO ClusteringUtils: Found 0 files in pending clustering operations 22/10/15 20:50:23 INFO AbstractTableFileSystemView: Building file system view for partition (column_stats) 22/10/15 20:50:23 DEBUG AbstractTableFileSystemView: #files found in partition (column_stats) =25, Time taken =32 22/10/15 20:50:23 DEBUG HoodieTableFileSystemView: Adding file-groups for partition :column_stats, #FileGroups=2 22/10/15 20:50:23 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=25, NumFileGroups=2, FileGroupsCreationTime=7, StoreTimeTaken=0 22/10/15 20:50:23 DEBUG AbstractTableFileSystemView: Time to load partition (column_stats) =39 22/10/15 20:50:23 DEBUG WriteMarkersFactory: Instantiated MarkerFiles with marker type: DIRECT 22/10/15 20:50:23 INFO DirectWriteMarkers: Creating Marker Path=s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10553_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:23 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10553_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:23 INFO DirectWriteMarkers: [direct] Created marker file s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10553_20221015083833526.hfile.marker.APPEND in 85 ms 22/10/15 20:50:23 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer 22/10/15 20:50:23 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.11_1-0-1 22/10/15 20:50:24 INFO BlockManager: Removing RDD 189 22/10/15 20:50:24 INFO BlockManager: Removing RDD 216 22/10/15 20:50:24 INFO BlockManager: Removing RDD 206 22/10/15 20:50:24 INFO BlockManager: Removing RDD 179 22/10/15 20:50:25 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10553', fileLen=-1} 22/10/15 20:50:25 ERROR BaseSparkCommitActionExecutor: Error upserting bucketType UPDATE for partition :4 org.apache.hudi.exception.HoodieAppendException: Failed while appending records to s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10553 at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:410) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244) ~[__app__.jar:0.11.1-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:327) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.Task.run(Task.scala:138) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342] Caused by: java.lang.IllegalStateException: Writing multiple records with same key not supported for org.apache.hudi.common.table.log.block.HoodieHFileDataBlock at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.serializeRecords(HoodieHFileDataBlock.java:136) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:131) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:158) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:404) ~[__app__.jar:0.11.1-amzn-0] ... 31 more 22/10/15 20:50:25 WARN BlockManager: Putting block rdd_243_4 failed due to exception org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :4. 22/10/15 20:50:25 WARN BlockManager: Block rdd_243_4 could not be removed as it was not found on disk or in memory 22/10/15 20:50:25 ERROR Executor: Exception in task 4.5 in stage 103.0 (TID 10553) org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :4 at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:329) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244) ~[__app__.jar:0.11.1-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:327) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.Task.run(Task.scala:138) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342] Caused by: org.apache.hudi.exception.HoodieAppendException: Failed while appending records to s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10553 at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:410) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] ... 28 more Caused by: java.lang.IllegalStateException: Writing multiple records with same key not supported for org.apache.hudi.common.table.log.block.HoodieHFileDataBlock at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.serializeRecords(HoodieHFileDataBlock.java:136) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:131) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:158) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:404) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] ... 28 more 22/10/15 20:50:34 INFO YarnCoarseGrainedExecutorBackend: Got assigned task 10561 22/10/15 20:50:34 INFO Executor: Running task 4.9 in stage 103.0 (TID 10561) 22/10/15 20:50:34 INFO YarnCoarseGrainedExecutorBackend: Got assigned task 10562 22/10/15 20:50:34 INFO Executor: Running task 5.9 in stage 103.0 (TID 10562) 22/10/15 20:50:34 INFO ShuffleBlockFetcherIterator: Getting 10 (7.9 MiB) non-empty blocks including 3 (2.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 7 (5.5 MiB) remote blocks 22/10/15 20:50:34 INFO ShuffleBlockFetcherIterator: Started 2 remote fetches in 0 ms 22/10/15 20:50:34 INFO ShuffleBlockFetcherIterator: Getting 10 (7.9 MiB) non-empty blocks including 3 (2.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 7 (5.5 MiB) remote blocks 22/10/15 20:50:34 INFO ShuffleBlockFetcherIterator: Started 2 remote fetches in 0 ms 22/10/15 20:50:36 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20221015204237661 for file col-stats-0001 22/10/15 20:50:36 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY 22/10/15 20:50:36 INFO FileSystemViewManager: Creating in-memory based Table View 22/10/15 20:50:36 INFO FileSystemViewManager: Creating InMemory based view for basePath s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata 22/10/15 20:50:36 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups 22/10/15 20:50:36 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20221015204237661 for file col-stats-0000 22/10/15 20:50:36 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY 22/10/15 20:50:36 INFO FileSystemViewManager: Creating in-memory based Table View 22/10/15 20:50:36 INFO FileSystemViewManager: Creating InMemory based view for basePath s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata 22/10/15 20:50:36 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups 22/10/15 20:50:36 INFO ClusteringUtils: Found 0 files in pending clustering operations 22/10/15 20:50:36 INFO AbstractTableFileSystemView: Building file system view for partition (column_stats) 22/10/15 20:50:36 INFO ClusteringUtils: Found 0 files in pending clustering operations 22/10/15 20:50:36 INFO AbstractTableFileSystemView: Building file system view for partition (column_stats) 22/10/15 20:50:36 DEBUG AbstractTableFileSystemView: #files found in partition (column_stats) =25, Time taken =30 22/10/15 20:50:36 DEBUG HoodieTableFileSystemView: Adding file-groups for partition :column_stats, #FileGroups=2 22/10/15 20:50:36 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=25, NumFileGroups=2, FileGroupsCreationTime=7, StoreTimeTaken=0 22/10/15 20:50:36 DEBUG AbstractTableFileSystemView: Time to load partition (column_stats) =37 22/10/15 20:50:36 DEBUG AbstractTableFileSystemView: #files found in partition (column_stats) =25, Time taken =36 22/10/15 20:50:36 DEBUG WriteMarkersFactory: Instantiated MarkerFiles with marker type: DIRECT 22/10/15 20:50:36 DEBUG HoodieTableFileSystemView: Adding file-groups for partition :column_stats, #FileGroups=2 22/10/15 20:50:36 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=25, NumFileGroups=2, FileGroupsCreationTime=6, StoreTimeTaken=0 22/10/15 20:50:36 DEBUG AbstractTableFileSystemView: Time to load partition (column_stats) =42 22/10/15 20:50:36 DEBUG WriteMarkersFactory: Instantiated MarkerFiles with marker type: DIRECT 22/10/15 20:50:36 INFO DirectWriteMarkers: Creating Marker Path=s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0000_5-103-10562_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:36 INFO DirectWriteMarkers: Creating Marker Path=s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10561_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:36 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0000_5-103-10562_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:36 INFO MultipartUploadOutputStream: close closed:false s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10561_20221015083833526.hfile.marker.APPEND 22/10/15 20:50:36 INFO DirectWriteMarkers: [direct] Created marker file s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0000_5-103-10562_20221015083833526.hfile.marker.APPEND in 85 ms 22/10/15 20:50:36 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer 22/10/15 20:50:36 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0000_20221015083833526.log.11_1-0-1 22/10/15 20:50:36 INFO DirectWriteMarkers: [direct] Created marker file s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/.hoodie/.temp/20221015204237661/column_stats/col-stats-0001_4-103-10561_20221015083833526.hfile.marker.APPEND in 84 ms 22/10/15 20:50:36 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer 22/10/15 20:50:36 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.11_1-0-1 22/10/15 20:50:37 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0000_20221015083833526.log.12_5-103-10562', fileLen=-1} 22/10/15 20:50:37 ERROR BaseSparkCommitActionExecutor: Error upserting bucketType UPDATE for partition :5 org.apache.hudi.exception.HoodieAppendException: Failed while appending records to s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0000_20221015083833526.log.12_5-103-10562 at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:410) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244) ~[__app__.jar:0.11.1-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:327) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.Task.run(Task.scala:138) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342] Caused by: java.lang.IllegalStateException: Writing multiple records with same key not supported for org.apache.hudi.common.table.log.block.HoodieHFileDataBlock at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.serializeRecords(HoodieHFileDataBlock.java:136) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:131) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:158) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:404) ~[__app__.jar:0.11.1-amzn-0] ... 31 more 22/10/15 20:50:37 WARN BlockManager: Putting block rdd_243_5 failed due to exception org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :5. 22/10/15 20:50:37 WARN BlockManager: Block rdd_243_5 could not be removed as it was not found on disk or in memory 22/10/15 20:50:37 ERROR Executor: Exception in task 5.9 in stage 103.0 (TID 10562) org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :5 at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:329) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244) ~[__app__.jar:0.11.1-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:327) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.Task.run(Task.scala:138) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342] Caused by: org.apache.hudi.exception.HoodieAppendException: Failed while appending records to s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0000_20221015083833526.log.12_5-103-10562 at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:410) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] ... 28 more Caused by: java.lang.IllegalStateException: Writing multiple records with same key not supported for org.apache.hudi.common.table.log.block.HoodieHFileDataBlock at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.serializeRecords(HoodieHFileDataBlock.java:136) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:131) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:158) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:404) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] ... 28 more 22/10/15 20:50:37 INFO Executor: Executor is trying to kill task 4.9 in stage 103.0 (TID 10561), reason: Stage cancelled 22/10/15 20:50:37 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10561', fileLen=-1} 22/10/15 20:50:37 ERROR BaseSparkCommitActionExecutor: Error upserting bucketType UPDATE for partition :4 org.apache.hudi.exception.HoodieAppendException: Failed while appending records to s3://prod-signal-articles-store/articles_hudi_copy_on_write/.hoodie/metadata/column_stats/.col-stats-0001_20221015083833526.log.12_4-103-10561 at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:410) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:382) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:84) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244) ~[__app__.jar:0.11.1-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:327) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.scheduler.Task.run(Task.scala:138) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0-amzn-0.jar:3.3.0-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342] Caused by: java.lang.IllegalStateException: Writing multiple records with same key not supported for org.apache.hudi.common.table.log.block.HoodieHFileDataBlock at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.serializeRecords(HoodieHFileDataBlock.java:136) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:131) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:158) ~[__app__.jar:0.11.1-amzn-0] at org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:404) ~[__app__.jar:0.11.1-amzn-0] ... 31 more 22/10/15 20:50:37 WARN BlockManager: Putting block rdd_243_4 failed due to exception org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :4. 22/10/15 20:50:37 WARN BlockManager: Block rdd_243_4 could not be removed as it was not found on disk or in memory 22/10/15 20:50:37 INFO Executor: Executor interrupted and killed task 4.9 in stage 103.0 (TID 10561), reason: Stage cancelled 22/10/15 20:50:38 INFO BlockManager: Removing RDD 233 22/10/15 20:50:38 INFO BlockManager: Removing RDD 45 22/10/15 20:50:38 INFO BlockManager: Removing RDD 35 22/10/15 20:50:38 INFO BlockManager: Removing RDD 243 22/10/15 20:56:31 INFO BlockManager: Removing RDD 233 22/10/15 20:56:31 INFO BlockManager: Removing RDD 243 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
