logesr commented on issue #12685: URL: https://github.com/apache/hudi/issues/12685#issuecomment-3204652236
@ad1happy2go It is happening in Doing partition and writing data: user_click_stream_events [count at HoodieSparkSqlWriter.scala:706]. We also noticed that these prolonged tasks are taking exactly 17 mins which brings the suspicion of any timeout after the process hangs in between. Here are the executor logs for the task: 2025-08-19 15:12:33,929 INFO [dispatcher-Executor] executor.CoarseGrainedExecutorBackend (Logging.scala:logInfo(61)): Got assigned task 5181 2025-08-19 15:12:33,929 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] executor.Executor (Logging.scala:logInfo(61)): Running task 2.0 in stage 56.0 (TID 5174) 2025-08-19 15:12:34,049 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] storage.ShuffleBlockFetcherIterator (Logging.scala:logInfo(61)): Getting 89 (102.6 KiB) non-empty blocks including 12 (14.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 77 (88.5 KiB) remote blocks 2025-08-19 15:12:34,049 INFO [Executor task launch worker for task 18.0 in stage 56.0 (TID 5181)] storage.ShuffleBlockFetcherIterator (Logging.scala:logInfo(61)): Started 6 remote fetches in 0 ms 2025-08-19 15:12:34,049 INFO [Executor task launch worker for task 52.0 in stage 56.0 (TID 5009)] storage.ShuffleBlockFetcherIterator (Logging.scala:logInfo(61)): Getting 5 (4.2 KiB) non-empty blocks including 1 (868.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 4 (3.3 KiB) remote blocks 2025-08-19 15:12:34,050 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] storage.ShuffleBlockFetcherIterator (Logging.scala:logInfo(61)): Started 6 remote fetches in 1 ms 2025-08-19 15:12:34,106 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] view.FileSystemViewManager (FileSystemViewManager.java:createViewManager(245)): Creating View Manager with storage type :REMOTE_FIRST 2025-08-19 15:12:34,106 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] view.FileSystemViewManager (FileSystemViewManager.java:createViewManager(265)): Creating remote first table view 2025-08-19 15:12:34,106 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] view.FileSystemViewManager (FileSystemViewManager.java:createRemoteFileSystemView(214)): Creating remote view for basePath s3://<bucket_name>/events. Server=172.35.164.108:46525, Timeout=300 2025-08-19 15:12:34,106 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] view.FileSystemViewManager (FileSystemViewManager.java:createInMemoryFileSystemView(165)): Creating InMemory based view for basePath s3://<bucket_name>/events 2025-08-19 15:12:34,106 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] view.AbstractTableFileSystemView (AbstractTableFileSystemView.java:resetFileGroupsReplaced(244)): Took 0 ms to read 0 instants, 0 replaced file groups 2025-08-19 15:12:34,147 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] util.ClusteringUtils (ClusteringUtils.java:getAllFileGroupsInPendingClusteringPlans(137)): Found 0 files in pending clustering operations 2025-08-19 15:12:34,147 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] view.RemoteHoodieTableFileSystemView (RemoteHoodieTableFileSystemView.java:executeRequest(180)): Sending request : (http://172.35.164.108:46525/v1/hoodie/view/datafile/latest/partition?partition=event_edition_id%3Deve-b058582b-882a-4b01-a984-8cc4c75bc592&basepath=s3%3A%2F%2F<bucket_name>%2Fevents&fileid=6fed2e9b-bf37-4dae-9451-8968665c5710-0&lastinstantts=20250819142936912&timelinehash=db922ca4c541b7873f7fd4b5fef51fc4e480ff3b96869b1354aada33226139c8) 2025-08-19 15:12:34,161 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] io.HoodieMergeHandle (HoodieMergeHandle.java:initializeIncomingRecordsMap(230)): MaxMemoryPerPartitionMerge => 1073741824 2025-08-19 15:12:34,181 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] collection.ExternalSpillableMap (ExternalSpillableMap.java:put(210)): Estimated Payload size => 1992 2025-08-19 15:12:34,181 INFO [Executor task launch worker for task 256.0 in stage 56.0 (TID 5149)] collection.ExternalSpillableMap (ExternalSpillableMap.java:put(210)): Estimated Payload size => 1776 2025-08-19 15:12:34,184 INFO [Executor task launch worker for task 288.0 in stage 56.0 (TID 5169)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-4fe201cf-cfd4-47e0-bd4c-897fb3585d5d%2F5b366bcb-6bc8-4087-ac29-7b8271a8b3d5-0_288-56-5169_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,184 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] collection.ExternalSpillableMap (ExternalSpillableMap.java:put(220)): New Estimated Payload size => 1681 2025-08-19 15:12:34,184 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(263)): Number of entries in MemoryBasedMap => 109, Total size in bytes of MemoryBasedMap => 183257, Number of entries in BitCaskDiskMap => 0, Size of file spilled to disk => 0 2025-08-19 15:12:34,184 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(173)): partitionPath:event_edition_id=eve-b058582b-882a-4b01-a984-8cc4c75bc592, fileId to be merged:6fed2e9b-bf37-4dae-9451-8968665c5710-0 2025-08-19 15:12:34,187 INFO [Executor task launch worker for task 63.0 in stage 56.0 (TID 5202)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(173)): partitionPath:event_edition_id=eve-125eb6d7-acc7-4aa7-9db7-eaaf23091f56, fileId to be merged:56321a0c-6afe-442b-8c40-bc7a29d8ccc0-0 2025-08-19 15:12:34,190 INFO [Executor task launch worker for task 53.0 in stage 56.0 (TID 5016)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-33483351-f29c-4fc9-9462-a01c3e015843/ce8324ea-5f33-44d4-abe9-7c9d7406a975-0_49-52-5000_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-33483351-f29c-4fc9-9462-a01c3e015843/ce8324ea-5f33-44d4-abe9-7c9d7406a975-0_53-56-5016_20250819150738533.parquet 2025-08-19 15:12:34,191 INFO [Executor task launch worker for task 53.0 in stage 56.0 (TID 5016)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-33483351-f29c-4fc9-9462-a01c3e015843%2Fce8324ea-5f33-44d4-abe9-7c9d7406a975-0_53-56-5016_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,191 INFO [Executor task launch worker for task 28.0 in stage 56.0 (TID 4988)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-316cdb03-2fe1-4e2d-88d3-96fd79842dc4/f7bb8b15-8295-49e2-9e58-74cea515ffed-0_28-52-4999_20250819120725356.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-316cdb03-2fe1-4e2d-88d3-96fd79842dc4/f7bb8b15-8295-49e2-9e58-74cea515ffed-0_28-56-4988_20250819150738533.parquet 2025-08-19 15:12:34,191 INFO [Executor task launch worker for task 66.0 in stage 56.0 (TID 5023)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-40d2ad1b-753f-4f9e-a58c-42fa691762e0/cc01bfe4-78a0-497e-ac61-41b060a711dd-0_64-52-5025_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-40d2ad1b-753f-4f9e-a58c-42fa691762e0/cc01bfe4-78a0-497e-ac61-41b060a711dd-0_66-56-5023_20250819150738533.parquet 2025-08-19 15:12:34,192 INFO [Executor task launch worker for task 28.0 in stage 56.0 (TID 4988)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-316cdb03-2fe1-4e2d-88d3-96fd79842dc4%2Ff7bb8b15-8295-49e2-9e58-74cea515ffed-0_28-56-4988_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,192 INFO [Executor task launch worker for task 66.0 in stage 56.0 (TID 5023)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-40d2ad1b-753f-4f9e-a58c-42fa691762e0%2Fcc01bfe4-78a0-497e-ac61-41b060a711dd-0_66-56-5023_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,192 INFO [Executor task launch worker for task 256.0 in stage 56.0 (TID 5149)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-447756c1-8919-4bdd-9178-d2fabef2c303/1f570ddb-a73c-438b-b34e-bee002b4744d-0_256-52-5130_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-447756c1-8919-4bdd-9178-d2fabef2c303/1f570ddb-a73c-438b-b34e-bee002b4744d-0_256-56-5149_20250819150738533.parquet 2025-08-19 15:12:34,192 INFO [Executor task launch worker for task 256.0 in stage 56.0 (TID 5149)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-447756c1-8919-4bdd-9178-d2fabef2c303%2F1f570ddb-a73c-438b-b34e-bee002b4744d-0_256-56-5149_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,192 INFO [Executor task launch worker for task 52.0 in stage 56.0 (TID 5009)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-5d406622-edc7-485e-9828-e780ec993adc/b5ce577b-20f3-4175-b48b-abf89467ce93-0_48-52-5013_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-5d406622-edc7-485e-9828-e780ec993adc/b5ce577b-20f3-4175-b48b-abf89467ce93-0_52-56-5009_20250819150738533.parquet 2025-08-19 15:12:34,193 INFO [Executor task launch worker for task 52.0 in stage 56.0 (TID 5009)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-5d406622-edc7-485e-9828-e780ec993adc%2Fb5ce577b-20f3-4175-b48b-abf89467ce93-0_52-56-5009_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,193 INFO [Executor task launch worker for task 180.0 in stage 56.0 (TID 5093)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-1f505731-de3b-4f84-b711-2a802c783680/109ee361-dfa5-441b-a605-6d10cc8c55ed-0_187-52-5141_20250819130708476.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-1f505731-de3b-4f84-b711-2a802c783680/109ee361-dfa5-441b-a605-6d10cc8c55ed-0_180-56-5093_20250819150738533.parquet 2025-08-19 15:12:34,193 INFO [Executor task launch worker for task 240.0 in stage 56.0 (TID 5135)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-f31be9b0-91d3-4df5-b2a3-429aff0b8cd3/c1d15b15-e66d-48c2-875c-ed851f2de576-0_240-52-5259_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-f31be9b0-91d3-4df5-b2a3-429aff0b8cd3/c1d15b15-e66d-48c2-875c-ed851f2de576-0_240-56-5135_20250819150738533.parquet 2025-08-19 15:12:34,193 INFO [Executor task launch worker for task 18.0 in stage 56.0 (TID 5181)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-b0277548-bbaa-4ac7-9b55-993a2c176118/18348d4f-320f-43ce-b941-c2807ae9a7c3-0_18-52-5177_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-b0277548-bbaa-4ac7-9b55-993a2c176118/18348d4f-320f-43ce-b941-c2807ae9a7c3-0_18-56-5181_20250819150738533.parquet 2025-08-19 15:12:34,194 INFO [Executor task launch worker for task 180.0 in stage 56.0 (TID 5093)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-1f505731-de3b-4f84-b711-2a802c783680%2F109ee361-dfa5-441b-a605-6d10cc8c55ed-0_180-56-5093_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,194 INFO [Executor task launch worker for task 49.0 in stage 56.0 (TID 5195)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-e2cfdde6-53eb-4869-8ff9-8486a461d92e/eaa19e9c-3b8d-4a47-b2c4-27c037c043fd-0_45-52-5191_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-e2cfdde6-53eb-4869-8ff9-8486a461d92e/eaa19e9c-3b8d-4a47-b2c4-27c037c043fd-0_49-56-5195_20250819150738533.parquet 2025-08-19 15:12:34,194 INFO [Executor task launch worker for task 240.0 in stage 56.0 (TID 5135)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-f31be9b0-91d3-4df5-b2a3-429aff0b8cd3%2Fc1d15b15-e66d-48c2-875c-ed851f2de576-0_240-56-5135_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,194 INFO [Executor task launch worker for task 18.0 in stage 56.0 (TID 5181)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-b0277548-bbaa-4ac7-9b55-993a2c176118%2F18348d4f-320f-43ce-b941-c2807ae9a7c3-0_18-56-5181_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,195 INFO [Executor task launch worker for task 49.0 in stage 56.0 (TID 5195)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-e2cfdde6-53eb-4869-8ff9-8486a461d92e%2Feaa19e9c-3b8d-4a47-b2c4-27c037c043fd-0_49-56-5195_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,196 INFO [Executor task launch worker for task 250.0 in stage 56.0 (TID 5142)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-1458df01-3169-403b-bd95-286f46b7e86d/e44a3044-9923-42a0-9031-fee883a69928-0_249-52-5148_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-1458df01-3169-403b-bd95-286f46b7e86d/e44a3044-9923-42a0-9031-fee883a69928-0_250-56-5142_20250819150738533.parquet 2025-08-19 15:12:34,196 INFO [Executor task launch worker for task 250.0 in stage 56.0 (TID 5142)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-1458df01-3169-403b-bd95-286f46b7e86d%2Fe44a3044-9923-42a0-9031-fee883a69928-0_250-56-5142_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,196 INFO [Executor task launch worker for task 33.0 in stage 56.0 (TID 5188)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-7cc7ce9b-8a4f-4e12-b725-223467dd85c8/2a4eb602-8d63-40e4-bee4-0e72e2b2dd15-0_29-52-5184_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-7cc7ce9b-8a4f-4e12-b725-223467dd85c8/2a4eb602-8d63-40e4-bee4-0e72e2b2dd15-0_33-56-5188_20250819150738533.parquet 2025-08-19 15:12:34,196 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-b058582b-882a-4b01-a984-8cc4c75bc592/6fed2e9b-bf37-4dae-9451-8968665c5710-0_2-52-5171_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-b058582b-882a-4b01-a984-8cc4c75bc592/6fed2e9b-bf37-4dae-9451-8968665c5710-0_2-56-5174_20250819150738533.parquet 2025-08-19 15:12:34,197 INFO [Executor task launch worker for task 33.0 in stage 56.0 (TID 5188)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-7cc7ce9b-8a4f-4e12-b725-223467dd85c8%2F2a4eb602-8d63-40e4-bee4-0e72e2b2dd15-0_33-56-5188_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,197 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-b058582b-882a-4b01-a984-8cc4c75bc592%2F6fed2e9b-bf37-4dae-9451-8968665c5710-0_2-56-5174_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,199 INFO [Executor task launch worker for task 116.0 in stage 56.0 (TID 5058)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-7f94398c-cb1a-4367-b40d-f55d804ae9de/7df51ee7-9732-4c7c-9162-92361265de0c-0_122-52-5098_20250819130708476.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-7f94398c-cb1a-4367-b40d-f55d804ae9de/7df51ee7-9732-4c7c-9162-92361265de0c-0_116-56-5058_20250819150738533.parquet 2025-08-19 15:12:34,199 INFO [Executor task launch worker for task 116.0 in stage 56.0 (TID 5058)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-7f94398c-cb1a-4367-b40d-f55d804ae9de%2F7df51ee7-9732-4c7c-9162-92361265de0c-0_116-56-5058_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,201 INFO [Executor task launch worker for task 279.0 in stage 56.0 (TID 5163)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-bc14fde0-67cb-4da5-8fca-2499124796b2/3e981ec5-8141-43ea-bbef-168f2abe40fb-0_283-52-5270_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-bc14fde0-67cb-4da5-8fca-2499124796b2/3e981ec5-8141-43ea-bbef-168f2abe40fb-0_279-56-5163_20250819150738533.parquet 2025-08-19 15:12:34,202 INFO [Executor task launch worker for task 279.0 in stage 56.0 (TID 5163)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-bc14fde0-67cb-4da5-8fca-2499124796b2%2F3e981ec5-8141-43ea-bbef-168f2abe40fb-0_279-56-5163_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,205 INFO [Executor task launch worker for task 63.0 in stage 56.0 (TID 5202)] io.HoodieMergeHandle (HoodieMergeHandle.java:init(189)): Merging new data into oldPath s3://<bucket_name>/events/event_edition_id=eve-125eb6d7-acc7-4aa7-9db7-eaaf23091f56/56321a0c-6afe-442b-8c40-bc7a29d8ccc0-0_59-52-5011_20250819140707105.parquet, as newPath s3://<bucket_name>/events/event_edition_id=eve-125eb6d7-acc7-4aa7-9db7-eaaf23091f56/56321a0c-6afe-442b-8c40-bc7a29d8ccc0-0_63-56-5202_20250819150738533.parquet 2025-08-19 15:12:34,206 INFO [Executor task launch worker for task 63.0 in stage 56.0 (TID 5202)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:executeRequestToTimelineServer(168)): Sending request : (http://172.35.164.108:46525/v1/hoodie/marker/create?markername=event_edition_id%3Deve-125eb6d7-acc7-4aa7-9db7-eaaf23091f56%2F56321a0c-6afe-442b-8c40-bc7a29d8ccc0-0_63-56-5202_20250819150738533.parquet.marker.MERGE&markerdirpath=s3%3A%2F%2F<bucket_name>%2Fevents%2F.hoodie%2F.temp%2F20250819150738533) 2025-08-19 15:12:34,504 INFO [Executor task launch worker for task 63.0 in stage 56.0 (TID 5202)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:create(151)): [timeline-server-based] Created marker file event_edition_id=eve-125eb6d7-acc7-4aa7-9db7-eaaf23091f56/56321a0c-6afe-442b-8c40-bc7a29d8ccc0-0_63-56-5202_20250819150738533.parquet.marker.MERGE in 298 ms 2025-08-19 15:12:34,504 INFO [Executor task launch worker for task 279.0 in stage 56.0 (TID 5163)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:create(151)): [timeline-server-based] Created marker file event_edition_id=eve-bc14fde0-67cb-4da5-8fca-2499124796b2/3e981ec5-8141-43ea-bbef-168f2abe40fb-0_279-56-5163_20250819150738533.parquet.marker.MERGE in 302 ms 2025-08-19 15:12:34,504 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:create(151)): [timeline-server-based] Created marker file event_edition_id=eve-b058582b-882a-4b01-a984-8cc4c75bc592/6fed2e9b-bf37-4dae-9451-8968665c5710-0_2-56-5174_20250819150738533.parquet.marker.MERGE in 307 ms 2025-08-19 15:12:34,504 INFO [Executor task launch worker for task 49.0 in stage 56.0 (TID 5195)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:create(151)): [timeline-server-based] Created marker file event_edition_id=eve-e2cfdde6-53eb-4869-8ff9-8486a461d92e/eaa19e9c-3b8d-4a47-b2c4-27c037c043fd-0_49-56-5195_20250819150738533.parquet.marker.MERGE in 309 ms 2025-08-19 15:12:34,504 INFO [Executor task launch worker for task 250.0 in stage 56.0 (TID 5142)] marker.TimelineServerBasedWriteMarkers (TimelineServerBasedWriteMarkers.java:create(151)): [timeline-server-based] Created marker file event_edition_id=eve-1458df01-3169-403b-bd95-286f46b7e86d/e44a3044-9923-42a0-9031-fee883a69928-0_250-56-5142_20250819150738533.parquet.marker.MERGE in 308 ms 2025-08-19 15:12:34,939 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(939)): Opening 's3://<bucket_name>/events/.hoodie/20250819140707105.commit' for reading 2025-08-19 15:12:35,179 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(939)): Opening 's3://<bucket_name>/events/event_edition_id=eve-b058582b-882a-4b01-a984-8cc4c75bc592/6fed2e9b-bf37-4dae-9451-8968665c5710-0_2-52-5171_20250819140707105.parquet' for reading 2025-08-19 15:13:43,696 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] s3n.MultipartUploadOutputStream (MultipartUploadOutputStream.java:close(421)): close closed:false s3://<bucket_name>/events/event_edition_id=eve-b058582b-882a-4b01-a984-8cc4c75bc592/6fed2e9b-bf37-4dae-9451-8968665c5710-0_2-56-5174_20250819150738533.parquet 2025-08-19 15:29:23,315 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] io.HoodieMergeHandle (HoodieMergeHandle.java:close(435)): MergeHandle for partitionPath event_edition_id=eve-b058582b-882a-4b01-a984-8cc4c75bc592 fileID 6fed2e9b-bf37-4dae-9451-8968665c5710-0, took 1009154 ms. 2025-08-19 15:29:23,315 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] memory.MemoryStore (Logging.scala:logInfo(61)): Block rdd_155_2 stored as values in memory (estimated size 373.0 B, free 47.8 GiB) 2025-08-19 15:29:23,317 INFO [Executor task launch worker for task 2.0 in stage 56.0 (TID 5174)] executor.Executor (Logging.scala:logInfo(61)): Finished task 2.0 in stage 56.0 (TID 5174). 1147 bytes result sent to driver -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
