pravin1406 opened a new issue, #9826:
URL: https://github.com/apache/hudi/issues/9826
Hi,
In one of the cases when we writing incremental data (upsert) in a hudi
table, we failed to write deltacommit file on storage, which caused the
HoodieIOException. After which spark context exited. But the JVM did not die.
When we took a thread dump we non daemon many thread pools still running, which
would have caused the JVM to not shutdown.
Further upon debugging, we came to conclusion that this was due to marker
handler creating 2 set of threadpools in non daemon way. Also when the
exception was thrown, the timeline service thread was stopped as it was daemon
(we have made changes to open source code). But marker handler threads were
left non daemon. Had the exception was caught properly and .close of timeline
service was called in finally block, this would have not happened. I've tried
to make this these thread pool deamon. Also wanted to handle gracefull shutdown
of the timeline service. but couldn't figure it out. Need you help.
Attaching exception stacktrace and threadump.
**Environment Description**
* Hudi version : 0.12.2
* Spark version : 3.2.0
* Hive version : 3.1.1
* Hadoop version : 3.1.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
`Exception in thread "main" org.apache.hudi.exception.HoodieIOException:
Failed to create file
s3a://intsys-erp/datalake_staging/p_fa_books/.hoodie/metadata/.hoodie/20231002172521154.deltacommit.inflight
at
org.apache.hudi.common.util.FileIOUtils.createFileInPath(FileIOUtils.java:183)
at
org.apache.hudi.common.util.FileIOUtils.createFileInPath(FileIOUtils.java:189)
at
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:580)
at
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionRequestedToInflight(HoodieActiveTimeline.java:642)
at
org.apache.hudi.table.action.commit.BaseCommitActionExecutor.saveWorkloadProfileMetadataToInflight(BaseCommitActionExecutor.java:148)
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:163)
at
org.apache.hudi.table.action.deltacommit.SparkUpsertPreppedDeltaCommitActionExecutor.execute(SparkUpsertPreppedDeltaCommitActionExecutor.java:45)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:107)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:76)
at
org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:173)
at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:166)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:819)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:886)
at
org.apache.hudi.client.BaseHoodieWriteClient.lambda$writeTableMetadata$52(BaseHoodieWriteClient.java:350)
at
org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.client.BaseHoodieWriteClient.writeTableMetadata(BaseHoodieWriteClient.java:350)
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:282)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:235)
at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:126)
at
org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:701)
at
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:345)
at
org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at
org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:106)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:93)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:91)
at
org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:128)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
at
org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
at
in.airtel.di.ingestion.spark.sink.HudiSink.write(HudiSink.java:60)
at
in.airtel.di.ingestion.plan.PlanExecutor.executePlan(PlanExecutor.java:242)
at
in.airtel.di.ingestion.spark.workflow.IngestionManager.main(IngestionManager.java:133)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.fs.s3a.AWSBadRequestException: Writing Object
on
datalake_staging/p_fa_books/.hoodie/metadata/.hoodie/20231002172521154.deltacommit.inflight:
com.amazonaws.services.s3.model.AmazonS3Exception: The Content-MD5 you
specified did not match what we received. (Service: Amazon S3; Status Code:
400; Error Code: BadDigest; Request ID: 1696263119880677; S3 Extended Request
ID: null; Proxy: null), S3 Extended Request ID: null:BadDigest: The Content-MD5
you specified did not match what we received. (Service: Amazon S3; Status Code:
400; Error Code: BadDigest; Request ID: 1696263119880677; S3 Extended Request
ID: null; Proxy: null)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:243)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:320)
at
org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:316)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:291)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:168)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:515)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:548)
at
`
`
"pool-33-thread-1" #1209 prio=5 os_prio=0 cpu=73898.01ms elapsed=138383.49s
tid=0x00007f2d80011000 nid=0x510 waiting on condition [0x00007f2d089d8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a1f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown
Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-1" #1210 prio=5 os_prio=0 cpu=44.10ms elapsed=138383.44s
tid=0x00007f2b94001000 nid=0x511 waiting on condition [0x00007f2d088d7000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-2" #1214 prio=5 os_prio=0 cpu=30.87ms elapsed=138383.26s
tid=0x00007f2b94004000 nid=0x515 waiting on condition [0x00007f2d084d3000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-3" #1215 prio=5 os_prio=0 cpu=24.93ms elapsed=138383.26s
tid=0x00007f2b94005800 nid=0x516 waiting on condition [0x00007f2d083d2000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-4" #1216 prio=5 os_prio=0 cpu=25.34ms elapsed=138383.24s
tid=0x00007f2b94008000 nid=0x517 waiting on condition [0x00007f2d082d1000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-5" #1225 prio=5 os_prio=0 cpu=23.86ms elapsed=138383.19s
tid=0x00007f2b94008800 nid=0x520 waiting on condition [0x00007f2ca38f7000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-6" #1228 prio=5 os_prio=0 cpu=25.06ms elapsed=138383.14s
tid=0x00007f2b94009800 nid=0x523 waiting on condition [0x00007f2ca35f4000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-7" #1231 prio=5 os_prio=0 cpu=24.16ms elapsed=138383.04s
tid=0x00007f2b9400a800 nid=0x526 waiting on condition [0x00007f2ca32f1000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-8" #1234 prio=5 os_prio=0 cpu=22.51ms elapsed=138382.79s
tid=0x00007f2b9400c000 nid=0x529 waiting on condition [0x00007f2ca2fee000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-9" #1235 prio=5 os_prio=0 cpu=24.72ms elapsed=138382.74s
tid=0x00007f2b9400d800 nid=0x52a waiting on condition [0x00007f2ca2eed000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-10" #1240 prio=5 os_prio=0 cpu=30.55ms elapsed=138382.69s
tid=0x00007f2b9400f000 nid=0x52f waiting on condition [0x00007f2ca29e8000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-11" #1243 prio=5 os_prio=0 cpu=23.66ms elapsed=138382.64s
tid=0x00007f2b94010000 nid=0x532 waiting on condition [0x00007f2ca20df000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-12" #1246 prio=5 os_prio=0 cpu=24.58ms elapsed=138382.49s
tid=0x00007f2b94010800 nid=0x535 waiting on condition [0x00007f2ca1bda000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-13" #1249 prio=5 os_prio=0 cpu=22.96ms elapsed=138382.44s
tid=0x00007f2b94012000 nid=0x538 waiting on condition [0x00007f2ca12d1000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-14" #1251 prio=5 os_prio=0 cpu=23.26ms elapsed=138382.39s
tid=0x00007f2b94013800 nid=0x53a waiting on condition [0x00007f2ca0fce000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-15" #1255 prio=5 os_prio=0 cpu=23.24ms elapsed=138382.34s
tid=0x00007f2b94014800 nid=0x53e waiting on condition [0x00007f2ca0ac9000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-16" #1259 prio=5 os_prio=0 cpu=24.75ms elapsed=138382.29s
tid=0x00007f2b94016000 nid=0x542 waiting on condition [0x00007f2ca06c5000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-17" #1261 prio=5 os_prio=0 cpu=26.25ms elapsed=138382.24s
tid=0x00007f2b94017800 nid=0x544 waiting on condition [0x00007f2ca03c2000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-18" #1269 prio=5 os_prio=0 cpu=24.80ms elapsed=138382.14s
tid=0x00007f2b94018800 nid=0x54c waiting on condition [0x00007f2baf6f5000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-19" #1272 prio=5 os_prio=0 cpu=24.39ms elapsed=138382.09s
tid=0x00007f2b94019800 nid=0x54f waiting on condition [0x00007f2baf1f0000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pool-34-thread-20" #1275 prio=5 os_prio=0 cpu=22.26ms elapsed=138382.04s
tid=0x00007f2b9401c000 nid=0x552 waiting on condition [0x00007f2bae7e6000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c629a3d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown
Source)
at
java.util.concurrent.LinkedBlockingQueue.take([email protected]/Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"qtp2055900787-1287" #1287 daemon prio=5 os_prio=0 cpu=190.41ms
elapsed=138381.78s tid=0x00007f2d44003800 nid=0x55e waiting on condition
[0x00007f2bacfd0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000006c62aef90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown
Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/Unknown
Source)
at
org.apache.hudi.org.apache.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at
org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)
at
org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018)
at java.lang.Thread.run([email protected]/Unknown Source)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
