zhangyue19921010 opened a new pull request, #10119:
URL: https://github.com/apache/hudi/pull/10119
### Change Logs
If Flink is crashed after instant committed but before
`this.ckpMetadata.commitInstant(instant);`
Re-start, Flink may re-use a committed instant in append mode.
Will check candidate re-used instant is committed or not
if committed, Hoodie would abort current instant.
### Impact
Flink append mode.
### Risk level (write none, low medium or high below)
low
Committed instant 20231114121907373 is re-used in next writing loop.
related logs
```
2023-11-14 12:20:04.791 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init write metadata
state : WriteMetadataEvent{writeStatusesSize=1, taskID=524,
instantTime='20231114121907373', lastBatch=false, endInput=false,
bootstrap=true} &
2023-11-14 12:20:04.792 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init write metadata
state : WriteMetadataEvent{writeStatusesSize=1, taskID=536,
instantTime='20231114121907373', lastBatch=false, endInput=false,
bootstrap=true} &
2023-11-14 12:20:04.792 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init write metadata
state : WriteMetadataEvent{writeStatusesSize=1, taskID=121,
instantTime='20231114121907373', lastBatch=false, endInput=false,
bootstrap=true} &
2023-11-14 12:20:04.792 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init write metadata
state : WriteMetadataEvent{writeStatusesSize=1, taskID=676,
instantTime='20231114121907373', lastBatch=false, endInput=false,
bootstrap=true} &
2023-11-14 12:20:04.792 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init write metadata
state : WriteMetadataEvent{writeStatusesSize=1, taskID=335,
instantTime='20231114121907373', lastBatch=false, endInput=false,
bootstrap=true} &
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init state
currentInstant = 20231114121907373
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init state
currentInstant = 20231114121907373
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init state
currentInstant = 20231114121907373
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init state
currentInstant = 20231114121907373
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Init state
currentInstant = 20231114121907373
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Restore write metadata
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Restore write metadata
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Restore write metadata
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Restore write metadata
2023-11-14 12:20:04.835 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Restore write metadata
2023-11-14 12:20:04.888 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:04.888 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:04.888 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:04.888 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:04.888 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:04.889 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Send bootstrap event
when restore write metadata
2023-11-14 12:20:04.890 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Send bootstrap event
when restore write metadata
2023-11-14 12:20:04.890 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.append.AppendWriteFunction - Recover task[536] for instant
[20231114121907373] with attemptId [114]
2023-11-14 12:20:04.889 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Send bootstrap event
when restore write metadata
2023-11-14 12:20:04.889 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Send bootstrap event
when restore write metadata
2023-11-14 12:20:04.889 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Send bootstrap event
when restore write metadata
2023-11-14 12:20:04.890 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.append.AppendWriteFunction - Recover task[121] for instant
[20231114121907373] with attemptId [114]
2023-11-14 12:20:04.890 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.append.AppendWriteFunction - Recover task[676] for instant
[20231114121907373] with attemptId [114]
2023-11-14 12:20:04.890 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.append.AppendWriteFunction - Recover task[524] for instant
[20231114121907373] with attemptId [114]
2023-11-14 12:20:04.890 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.append.AppendWriteFunction - Recover task[335] for instant
[20231114121907373] with attemptId [114]
2023-11-14 12:20:04.941 INFO [Source Data Fetcher :2/160)#114]
org.apache.kafka.clients.Metadata - [Consumer
clientId=Hudi-Traffic-BDM1695731932233-121,
groupId=Hudi-Traffic-BDM1695731932233] Cluster ID: hjnIVBMqQpiRVldDgpNfZw
2023-11-14 12:20:04.941 INFO [Source Data Fetcher :2/160)#114]
org.apache.kafka.clients.Metadata - [Consumer
clientId=Hudi-Traffic-BDM1695731932233-121,
groupId=Hudi-Traffic-BDM1695731932233] Cluster ID: hjnIVBMqQpiRVldDgpNfZw
2023-11-14 12:20:04.945 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Execute
instantToWrite method : hasData=true, currentInstant=null
2023-11-14 12:20:04.945 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Execute
instantToWrite method : hasData=true, currentInstant=null
2023-11-14 12:20:04.945 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Execute
instantToWrite method : hasData=true, currentInstant=null
2023-11-14 12:20:04.945 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Execute
instantToWrite method : hasData=true, currentInstant=null
2023-11-14 12:20:04.945 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Execute
instantToWrite method : hasData=true, currentInstant=null
2023-11-14 12:20:04.987 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Find the new instant,
hasData=true, the instant is 20231114121907373
2023-11-14 12:20:04.987 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Find the new instant,
hasData=true, the instant is 20231114121907373
2023-11-14 12:20:04.987 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Find the new instant,
hasData=true, the instant is 20231114121907373
2023-11-14 12:20:04.987 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Find the new instant,
hasData=true, the instant is 20231114121907373
2023-11-14 12:20:04.987 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.common.AbstractStreamWriteFunction - Find the new instant,
hasData=true, the instant is 20231114121907373
2023-11-14 12:20:04.993 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading
HoodieTableMetaClient from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:04.993 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading
HoodieTableMetaClient from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:04.993 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading
HoodieTableMetaClient from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:04.993 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading
HoodieTableMetaClient from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:04.993 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading
HoodieTableMetaClient from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.031 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/hoodie.properties
2023-11-14 12:20:05.031 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/hoodie.properties
2023-11-14 12:20:05.031 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/hoodie.properties
2023-11-14 12:20:05.031 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/hoodie.properties
2023-11-14 12:20:05.031 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/hoodie.properties
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of
type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of
type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of
type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading Active commit
timeline for
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of
type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of
type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading Active commit
timeline for
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.057 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading Active commit
timeline for
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.057 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading Active commit
timeline for
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.056 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.HoodieTableMetaClient - Loading Active commit
timeline for
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log
2023-11-14 12:20:05.079 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:05.079 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:05.082 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:05.082 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:05.083 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20231114121907373__commit__COMPLETED]}
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating View Manager
with storage type :REMOTE_FIRST
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating View Manager
with storage type :REMOTE_FIRST
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating View Manager
with storage type :REMOTE_FIRST
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating View Manager
with storage type :REMOTE_FIRST
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating View Manager
with storage type :REMOTE_FIRST
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating remote first
table view
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating remote first
table view
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating remote first
table view
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating remote first
table view
2023-11-14 12:20:05.092 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.common.table.view.FileSystemViewManager - Creating remote first
table view
2023-11-14 12:20:05.104 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Creating new file for
partition path 2023-11-14
2023-11-14 12:20:05.104 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Creating new file for
partition path 2023-11-14
2023-11-14 12:20:05.104 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Creating new file for
partition path 2023-11-14
2023-11-14 12:20:05.104 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Creating new file for
partition path 2023-11-14
2023-11-14 12:20:05.104 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Creating new file for
partition path 2023-11-14
2023-11-14 12:20:05.171 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - Creating Marker
Path=hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/ed777698-79ae-472c-8f25-c083347b113e-0_335-750-114_20231114121907373.parquet.marker.CREATE
2023-11-14 12:20:05.172 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - Creating Marker
Path=hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/4da81b8b-6be0-4e95-ba2d-8cfc559edef1-0_121-750-114_20231114121907373.parquet.marker.CREATE
2023-11-14 12:20:05.172 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - Creating Marker
Path=hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/0c60793b-3c69-4615-9357-6f4e45a0207e-0_536-750-114_20231114121907373.parquet.marker.CREATE
2023-11-14 12:20:05.172 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - Creating Marker
Path=hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/ebefc375-1a1d-4fa5-aad4-2f2e5e294fa8-0_524-750-114_20231114121907373.parquet.marker.CREATE
2023-11-14 12:20:05.172 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - Creating Marker
Path=hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/0abb5654-dfeb-43b4-ab3b-1caad09b449f-0_676-750-114_20231114121907373.parquet.marker.CREATE
2023-11-14 12:20:05.210 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - [direct] Created marker file
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/0c60793b-3c69-4615-9357-6f4e45a0207e-0_536-750-114_20231114121907373.parquet.marker.CREATE
in 57 ms
2023-11-14 12:20:05.210 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - [direct] Created marker file
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/ed777698-79ae-472c-8f25-c083347b113e-0_335-750-114_20231114121907373.parquet.marker.CREATE
in 57 ms
2023-11-14 12:20:05.210 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - [direct] Created marker file
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/ebefc375-1a1d-4fa5-aad4-2f2e5e294fa8-0_524-750-114_20231114121907373.parquet.marker.CREATE
in 57 ms
2023-11-14 12:20:05.210 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - [direct] Created marker file
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/4da81b8b-6be0-4e95-ba2d-8cfc559edef1-0_121-750-114_20231114121907373.parquet.marker.CREATE
in 57 ms
2023-11-14 12:20:05.210 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.table.marker.DirectWriteMarkers - [direct] Created marker file
hdfs://ns8/user/jdw_traffic/bdm_dev.db/hudi_bdm_m14_wireless_exposure_log/.hoodie/.temp/20231114121907373/2023-11-14/0abb5654-dfeb-43b4-ab3b-1caad09b449f-0_676-750-114_20231114121907373.parquet.marker.CREATE
in 57 ms
2023-11-14 12:20:05.599 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle - New handle created
for partition :2023-11-14 with fileId ebefc375-1a1d-4fa5-aad4-2f2e5e294fa8-0
2023-11-14 12:20:05.599 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle - New handle created
for partition :2023-11-14 with fileId ed777698-79ae-472c-8f25-c083347b113e-0
2023-11-14 12:20:05.599 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle - New handle created
for partition :2023-11-14 with fileId 4da81b8b-6be0-4e95-ba2d-8cfc559edef1-0
2023-11-14 12:20:05.599 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle - New handle created
for partition :2023-11-14 with fileId 0abb5654-dfeb-43b4-ab3b-1caad09b449f-0
2023-11-14 12:20:05.599 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle - New handle created
for partition :2023-11-14 with fileId 0c60793b-3c69-4615-9357-6f4e45a0207e-0
2023-11-14 12:21:00.624 INFO [Sink: hoodie_append_:2/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Closing bulk insert file
4da81b8b-6be0-4e95-ba2d-8cfc559edef1-0_121-750-114_20231114121907373.parquet
2023-11-14 12:21:00.624 INFO [Sink: hoodie_append_:6/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Closing bulk insert file
ed777698-79ae-472c-8f25-c083347b113e-0_335-750-114_20231114121907373.parquet
2023-11-14 12:21:00.624 INFO [Sink: hoodie_append_:5/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Closing bulk insert file
ebefc375-1a1d-4fa5-aad4-2f2e5e294fa8-0_524-750-114_20231114121907373.parquet
2023-11-14 12:21:00.624 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Closing bulk insert file
0c60793b-3c69-4615-9357-6f4e45a0207e-0_536-750-114_20231114121907373.parquet
2023-11-14 12:21:00.625 INFO [Sink: hoodie_append_:7/750)#114]
org.apache.hudi.sink.bulk.BulkInsertWriterHelper - Closing bulk insert file
0abb5654-dfeb-43b4-ab3b-1caad09b449f-0_676-750-114_20231114121907373.parquet
```
```
2023-11-14 12:20:00.293 INFO [pool-86-thread-1:6-thread-1]
org.apache.hudi.client.BaseHoodieWriteClient - Committing 20231114121907373
action commit
2023-11-14 12:20:00.340 INFO [pool-86-thread-1:6-thread-1]
org.apache.hudi.client.BaseHoodieWriteClient - Committing 20231114121907373
action commit
```
```
```
### Documentation Update
_Describe any necessary documentation update if there is any new feature,
config, or user-facing change_
- _The config description must be updated if new configs are added or the
default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website.
Please create a Jira ticket, attach the
ticket number here and follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to
make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
