xiaokxluoshu opened a new issue, #7046:
URL: https://github.com/apache/hudi/issues/7046
**Environment Description**
* Hudi version : 0.12.0
* Spark version : 2.4
* Flink version: 1.13.6
* Hive version :
* Hadoop version : 3.2.2
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : NO
**Describe the problem you faced**
We use flink to write hudi and use table.type=cow to write 20 million data
and then stop. After 30 minutes, we write again
````
2022-10-24 11:43:52,599 INFO
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner [] -
MaxMemoryInBytes allowed for compaction => 1073741824
2022-10-24 11:43:52,599 INFO
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner [] - Number of
entries in MemoryBasedMap in ExternalSpillableMap => 2
2022-10-24 11:43:52,599 INFO
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner [] - Total size
in bytes of MemoryBasedMap in ExternalSpillableMap => 8656
2022-10-24 11:43:52,599 INFO
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner [] - Number of
entries in BitCaskDiskMap in ExternalSpillableMap => 0
2022-10-24 11:43:52,599 INFO
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner [] - Size of file
spilled to disk => 0
2022-10-24 11:43:52,600 INFO org.apache.hudi.io.HoodieMergeHandle
[] - partitionPath:files, fileId to be merged:files-0000
2022-10-24 11:43:52,603 INFO org.apache.hudi.io.HoodieMergeHandle
[] - Merging new data into oldPath
hdfs://hadoop01:9000/hudi/model-series-intact-data/.hoodie/metadata/files/files-0000_0-1-0_20221024100338853001.hfile,
as newPath
hdfs://hadoop01:9000/hudi/model-series-intact-data/.hoodie/metadata/files/files-0000_0-1-0_20221024114333232001.hfile
2022-10-24 11:43:52,664 INFO
org.apache.hudi.table.marker.DirectWriteMarkers [] - Creating
Marker
Path=hdfs://hadoop01:9000/hudi/model-series-intact-data/.hoodie/metadata/.hoodie/.temp/20221024114333232001/files/files-0000_0-1-0_20221024114333232001.hfile.marker.MERGE
2022-10-24 11:43:52,674 INFO
org.apache.hudi.table.marker.DirectWriteMarkers [] - [direct]
Created marker file
hdfs://hadoop01:9000/hudi/model-series-intact-data/.hoodie/metadata/.hoodie/.temp/20221024114333232001/files/files-0000_0-1-0_20221024114333232001.hfile.marker.MERGE
in 71 ms
2022-10-24 11:43:52,681 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new compressor [.gz]
2022-10-24 11:43:52,682 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new compressor [.gz]
2022-10-24 11:43:52,689 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new decompressor [.gz]
2022-10-24 11:43:52,689 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new decompressor [.gz]
2022-10-24 11:43:52,689 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new decompressor [.gz]
2022-10-24 11:43:52,690 INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer [] - starting to
buffer records
2022-10-24 11:43:52,691 INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor [] - starting
consumer thread
2022-10-24 11:43:52,695 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new decompressor [.gz]
2022-10-24 11:43:52,697 INFO
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer [] - finished
buffering records
2022-10-24 11:43:52,699 INFO
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor [] - Queue
Consumption is done; notifying producer threads
2022-10-24 11:43:52,700 ERROR
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [initialize instant 20221024114339630] error
java.lang.NoSuchMethodError:
org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
at
org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:249)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:296)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.closeStreams(HFileBlock.java:1825)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFilePreadReader.close(HFilePreadReader.java:107)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.close(HFileReaderImpl.java:1421)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.io.storage.HoodieHFileReader.close(HoodieHFileReader.java:218)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.action.commit.FlinkMergeHelper.runMerge(FlinkMergeHelper.java:107)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdateInternal(HoodieFlinkCopyOnWriteTable.java:380)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdate(HoodieFlinkCopyOnWriteTable.java:371)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:227)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.action.compact.HoodieCompactor.lambda$compact$57154431$1(HoodieCompactor.java:138)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.common.function.FunctionWrapper.lambda$throwingMapWrapper$0(FunctionWrapper.java:38)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_261]
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
~[?:1.8.0_261]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_261]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_261]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
~[?:1.8.0_261]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_261]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
~[?:1.8.0_261]
at
java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_261]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_261]
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
~[?:1.8.0_261]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
~[?:1.8.0_261]
at
java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_261]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_261]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_261]
at
org.apache.hudi.common.data.HoodieBaseListData.<init>(HoodieBaseListData.java:41)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.common.data.HoodieListData.<init>(HoodieListData.java:68)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.common.data.HoodieListData.map(HoodieListData.java:105)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:138)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:88)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.compact(HoodieFlinkMergeOnReadTable.java:118)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.compact(HoodieFlinkWriteClient.java:429)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.client.BaseHoodieWriteClient.lambda$runAnyPendingCompactions$1(BaseHoodieWriteClient.java:592)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
~[?:1.8.0_261]
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
~[?:1.8.0_261]
at
org.apache.hudi.client.BaseHoodieWriteClient.runAnyPendingCompactions(BaseHoodieWriteClient.java:590)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.client.BaseHoodieWriteClient.runAnyPendingCompactions(BaseHoodieWriteClient.java:641)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.compactIfNecessary(HoodieBackedTableMetadataWriter.java:1007)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:115)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:813)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:880)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.writeTableMetadata(HoodieFlinkWriteClient.java:281)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:271)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:236)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:117)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:530)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:506)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:477)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$initInstant$6(StreamWriteOperatorCoordinator.java:399)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[blob_p-85bf6a9f06cb91bd20775cc5cfd6a95060810396-32878570929da3966468820ac5787a7f:0.12.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_261]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
2022-10-24 11:43:52,703 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'hoodie_append_write: data_hudi_sink' (operator
20600d0a6cfd8abe54715a1cd220ef3f).
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:553)
~[flink-dist_2.11-1.13.6.jar:1.13.6]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:187)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:146)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:133)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_261]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes
action [initialize instant 20221024114339630] error
... 6 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
at
org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:249)
~[?:?]
at
org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:296)
~[?:?]
at
org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.closeStreams(HFileBlock.java:1825)
~[?:?]
at
org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFilePreadReader.close(HFilePreadReader.java:107)
~[?:?]
at
org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.close(HFileReaderImpl.java:1421)
~[?:?]
at
org.apache.hudi.io.storage.HoodieHFileReader.close(HoodieHFileReader.java:218)
~[?:?]
at
org.apache.hudi.table.action.commit.FlinkMergeHelper.runMerge(FlinkMergeHelper.java:107)
~[?:?]
at
org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdateInternal(HoodieFlinkCopyOnWriteTable.java:380)
~[?:?]
at
org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdate(HoodieFlinkCopyOnWriteTable.java:371)
~[?:?]
at
org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:227)
~[?:?]
at
org.apache.hudi.table.action.compact.HoodieCompactor.lambda$compact$57154431$1(HoodieCompactor.java:138)
~[?:?]
at
org.apache.hudi.common.function.FunctionWrapper.lambda$throwingMapWrapper$0(FunctionWrapper.java:38)
~[?:?]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_261]
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
~[?:1.8.0_261]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_261]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_261]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
~[?:1.8.0_261]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
~[?:1.8.0_261]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
~[?:1.8.0_261]
at
java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
~[?:1.8.0_261]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_261]
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
~[?:1.8.0_261]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
~[?:1.8.0_261]
at
java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
~[?:1.8.0_261]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
~[?:1.8.0_261]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[?:1.8.0_261]
at
org.apache.hudi.common.data.HoodieBaseListData.<init>(HoodieBaseListData.java:41)
~[?:?]
at
org.apache.hudi.common.data.HoodieListData.<init>(HoodieListData.java:68) ~[?:?]
at
org.apache.hudi.common.data.HoodieListData.map(HoodieListData.java:105) ~[?:?]
at
org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:138)
~[?:?]
at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:88)
~[?:?]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.compact(HoodieFlinkMergeOnReadTable.java:118)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.compact(HoodieFlinkWriteClient.java:429)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.lambda$runAnyPendingCompactions$1(BaseHoodieWriteClient.java:592)
~[?:?]
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
~[?:1.8.0_261]
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
~[?:1.8.0_261]
at
org.apache.hudi.client.BaseHoodieWriteClient.runAnyPendingCompactions(BaseHoodieWriteClient.java:590)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.runAnyPendingCompactions(BaseHoodieWriteClient.java:641)
~[?:?]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.compactIfNecessary(HoodieBackedTableMetadataWriter.java:1007)
~[?:?]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:115)
~[?:?]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:813)
~[?:?]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:880)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.writeTableMetadata(HoodieFlinkWriteClient.java:281)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:271)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:236)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:117)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:530)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:506)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:477)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$initInstant$6(StreamWriteOperatorCoordinator.java:399)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[?:?]
... 3 more
````
I found that the hadoop version used in hudi source code is 2.10.1,
resulting in the hadoop-hdfs-client version is 2.10.1, However, I use hadoop
3.2.2 and the version of hadoop-hdfs-client is 3.2.2 in the flink automatic
import package `hadoop classpath`. In this case, the above exception will
occur. Is there any solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
