ns-shua commented on issue #32508:
URL: https://github.com/apache/beam/issues/32508#issuecomment-2365443180
@liferoad And in diagnostic errors table I see a lot of
```
WARNING 2024-09-22T03:32:15.969Z Operation ongoing in bundle for at least
10m00s without completing Processing times in each step(millis) Step name:
WriteToCommonAndTenantProject/StorageApiLoads/Convert/Convert to message Time
spent in this step: IntSummaryStatistics{count=2412, sum=107, min=0,
average=0.044362, max=21} Step name:
WriteToCommonAndTenantProjectFailedInsert/StorageApiLoads/StorageApiWriteInconsistent/Write
Records Time spent in this step: IntSummaryStatistics{count=1934, sum=161,
min=0, average=0.083247, max=85} Step name:
WriteToCommonAndTenantProjectFailedInsert/PrepareWrite/ParDo(Anonymous) Time
spent in this step: IntSummaryStatistics{count=1934, sum=0, min=0,
average=0.000000, max=0} Step name: ExtractFeature Time spent in this step:
IntSummaryStatistics{count=7, sum=5, min=0, average=0.714286, max=2} Step name:
WriteToCommonAndTenantProject/StorageApiLoads/StorageApiWriteInconsistent/Write
Records Time spent in this step: IntSummaryStatistics{count=2412, sum=1
4, min=0, average=0.005804, max=8} Step name:
WriteToCommonAndTenantProject/StorageApiLoads/rewindowIntoGlobal/Window.Assign
Time spent in this step: IntSummaryStatistics{count=2412, sum=0, min=0,
average=0.000000, max=0} Step name:
WriteToCommonAndTenantProjectFailedInsert/StorageApiLoads/Convert/Convert to
message Time spent in this step: IntSummaryStatistics{count=1934, sum=7, min=0,
average=0.003619, max=4} Step name: EnrichTenantMetadata Time spent in this
step: IntSummaryStatistics{count=1206, sum=0, min=0, average=0.000000, max=0}
Step name: Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Time spent in this
step: IntSummaryStatistics{count=7, sum=0, min=0, average=0.000000, max=0} Step
name: Reshuffle.ViaRandomKey/Values/Values/Map Time spent in this step:
IntSummaryStatistics{count=7, sum=0, min=0, average=0.000000, max=0} Step name:
DropFields.Inner/Select.Fields/ParDo(Select) Time spent in this step:
IntSummaryStatistics{count=1206, sum=0, min=0, average=0.000000, max=0} S
tep name: GEFMessageDecode Time spent in this step:
IntSummaryStatistics{count=7, sum=12, min=0, average=1.714286, max=11} Step
name: WriteToCommonAndTenantProject/PrepareWrite/ParDo(Anonymous) Time spent in
this step: IntSummaryStatistics{count=2412, sum=0, min=0, average=0.000000,
max=0} Step name:
WriteToCommonAndTenantProjectFailedInsert/StorageApiLoads/rewindowIntoGlobal/Window.Assign
Time spent in this step: IntSummaryStatistics{count=1934, sum=0, min=0,
average=0.000000, max=0} Step name: Reshuffle.ViaRandomKey/Reshuffle/GroupBy…
{
"insertId": "7981515283196192541:164283:0:12850531",
"jsonPayload": {
"worker": "txn-log-v4-static-tenant--09211208-8ico-harness-7411",
"message": "Operation ongoing in bundle for at least 10m00s without
completing\nProcessing times in each step(millis)\nStep name:
WriteToCommonAndTenantProject/StorageApiLoads/Convert/Convert to message\nTime
spent in this step: IntSummaryStatistics{count=2412, sum=107, min=0,
average=0.044362, max=21}\nStep name:
WriteToCommonAndTenantProjectFailedInsert/StorageApiLoads/StorageApiWriteInconsistent/Write
Records\nTime spent in this step: IntSummaryStatistics{count=1934, sum=161,
min=0, average=0.083247, max=85}\nStep name:
WriteToCommonAndTenantProjectFailedInsert/PrepareWrite/ParDo(Anonymous)\nTime
spent in this step: IntSummaryStatistics{count=1934, sum=0, min=0,
average=0.000000, max=0}\nStep name: ExtractFeature\nTime spent in this step:
IntSummaryStatistics{count=7, sum=5, min=0, average=0.714286, max=2}\nStep
name:
WriteToCommonAndTenantProject/StorageApiLoads/StorageApiWriteInconsistent/Write
Records\nTime spent in this step: IntSummaryStatistics{count=2412, sum=14, m
in=0, average=0.005804, max=8}\nStep name:
WriteToCommonAndTenantProject/StorageApiLoads/rewindowIntoGlobal/Window.Assign\nTime
spent in this step: IntSummaryStatistics{count=2412, sum=0, min=0,
average=0.000000, max=0}\nStep name:
WriteToCommonAndTenantProjectFailedInsert/StorageApiLoads/Convert/Convert to
message\nTime spent in this step: IntSummaryStatistics{count=1934, sum=7,
min=0, average=0.003619, max=4}\nStep name: EnrichTenantMetadata\nTime spent in
this step: IntSummaryStatistics{count=1206, sum=0, min=0, average=0.000000,
max=0}\nStep name: Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable\nTime spent
in this step: IntSummaryStatistics{count=7, sum=0, min=0, average=0.000000,
max=0}\nStep name: Reshuffle.ViaRandomKey/Values/Values/Map\nTime spent in this
step: IntSummaryStatistics{count=7, sum=0, min=0, average=0.000000,
max=0}\nStep name: DropFields.Inner/Select.Fields/ParDo(Select)\nTime spent in
this step: IntSummaryStatistics{count=1206, sum=0, min=0, average=0.000000,
max=0}\nStep name: GEFMessageDecode\nTime spent in this step:
IntSummaryStatistics{count=7, sum=12, min=0, average=1.714286, max=11}\nStep
name: WriteToCommonAndTenantProject/PrepareWrite/ParDo(Anonymous)\nTime spent
in this step: IntSummaryStatistics{count=2412, sum=0, min=0, average=0.000000,
max=0}\nStep name:
WriteToCommonAndTenantProjectFailedInsert/StorageApiLoads/rewindowIntoGlobal/Window.Assign\nTime
spent in this step: IntSummaryStatistics{count=1934, sum=0, min=0,
average=0.000000, max=0}\nStep name:
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/ReadStream\nTime spent in this
step: IntSummaryStatistics{count=14, sum=0, min=0, average=0.000000, max=0}\n
at [email protected]/java.lang.Thread.sleep(Native Method)\n at
app//org.apache.beam.sdk.util.Sleeper$$Lambda$764/0x0000000800c70c40.sleep(Unknown
Source)\n at
app//org.apache.beam.sdk.util.BackOffUtils.next(BackOffUtils.java:48)\n at
app//org.apache.beam.sdk.io.gcp.bigquery.RetryManager.await(RetryManager.java:317)\n
at
app//org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn.flushAll(StorageApiWriteUnshardedRecords.java:1026)\n
at
app//org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn.flushIfNecessary(StorageApiWriteUnshardedRecords.java:994)\n
at
app//org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn.process(StorageApiWriteUnshardedRecords.java:1144)\n
at
app//org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn$DoFnInvoker.invokeProcessElement(Unknown
Source)\n",
"thread": "36",
"job": "2024-09-21_12_07_17-18246689280237981546",
"logger":
"org.apache.beam.runners.dataflow.worker.DataflowExecutionContext$DataflowExecutionStateTracker"
},
"resource": {
"type": "dataflow_step",
"labels": {
"region": "australia-southeast1",
"step_id": "",
"job_id": "2024-09-21_12_07_17-18246689280237981546",
"job_name": "txn-log-v4-static-tenant-filter-dynamicdestination",
"project_id": "project-data-au"
}
},
"timestamp": "2024-09-22T03:32:15.969Z",
"severity": "WARNING",
"labels": {
"dataflow.googleapis.com/log_type": "supportability",
"dataflow.googleapis.com/region": "australia-southeast1",
"dataflow.googleapis.com/job_id":
"2024-09-21_12_07_17-18246689280237981546",
"dataflow.googleapis.com/job_name":
"txn-log-v4-static-tenant-filter-dynamicdestination",
"compute.googleapis.com/resource_type": "instance",
"compute.googleapis.com/resource_name":
"txn-log-v4-static-tenant--09211208-8ico-harness-7411",
"compute.googleapis.com/resource_id": "7981515283196192541"
},
"logName":
"projects/project-data-au/logs/dataflow.googleapis.com%2Fworker",
"receiveTimestamp": "2024-09-22T03:32:25.025214266Z",
"errorGroups": [
{
"id": "CLTglOLFqKTiag"
}
]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
