szingerpeter opened a new issue, #13311:
URL: https://github.com/apache/hudi/issues/13311
**Describe the problem you faced**
Hy,
I have an issue with the built-in Cloudwatch Metrics reporter, where it's
failing to send some metrics to cloudwatch.
It seems to me as if the session has been stopped before sending the last
information.
**To Reproduce**
Run a write operation with metrics enabled to CloudWatch.
**Expected behavior**
No failure, all metrics sent successfully
**Environment Description**
* Hudi version : 0.14.1
* Spark version : 3.5
**Stacktrace**
```
25/04/15 08:30:14 INFO ApplicationMaster: Final app status: SUCCEEDED,
exitCode: 0
25/04/15 08:30:14 INFO SparkContext: Invoking stop() from shutdown hook
25/04/15 08:30:14 INFO SparkContext: SparkContext is stopping with exitCode
0.
25/04/15 08:30:14 INFO SparkUI: Stopped Spark web UI at <>
25/04/15 08:30:14 ERROR CloudWatchReporter: Error reporting metrics to
CloudWatch. The data in this CloudWatch request may have been discarded, and
not made it to CloudWatch.
java.util.concurrent.ExecutionException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: event executor terminated
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_432]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
~[?:1.8.0_432]
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234)
~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:211)
~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
at
org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
~[hudi-utilities-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at
org.apache.hudi.metrics.cloudwatch.CloudWatchMetricsReporter.report(CloudWatchMetricsReporter.java:71)
~[hudi-utilities-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_432]
at org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:118)
~[hudi-utilities-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_432]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable
to execute HTTP request: event executor terminated
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
~[?:1.8.0_432]
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
~[?:1.8.0_432]
at
java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
~[?:1.8.0_432]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.attemptExecute(AsyncRetryableStage.java:156)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:136)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.execute(AsyncRetryableStage.java:95)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:79)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:44)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:41)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:29)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:64)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:36)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:49)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:32)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:215)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(BaseAsyncClientHandler.java:288)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:227)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:80)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:294)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:73)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:49)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.services.cloudwatch.DefaultCloudWatchAsyncClient.putMetricData(DefaultCloudWatchAsyncClient.java:2601)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:229)
~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
... 6 more
Caused by: java.util.concurrent.RejectedExecutionException: event executor
terminated
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.lazyExecute0(SingleThreadEventExecutor.java:831)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.lazyExecute(SingleThreadEventExecutor.java:822)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:292)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:205)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutorGroup.schedule(AbstractEventExecutorGroup.java:50)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.http.nio.netty.internal.DelegatingEventLoopGroup.schedule(DelegatingEventLoopGroup.java:153)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.acquire(HealthCheckedChannelPool.java:67)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.http.nio.netty.internal.CancellableAcquireChannelPool.acquire(CancellableAcquireChannelPool.java:50)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.http.nio.netty.internal.SimpleChannelPoolAwareChannelPool.acquire(SimpleChannelPoolAwareChannelPool.java:42)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.acquireChannel(NettyRequestExecutor.java:104)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.execute(NettyRequestExecutor.java:93)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.execute(NettyNioAsyncHttpClient.java:123)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.doExecuteHttpRequest(MakeAsyncHttpRequestStage.java:204)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.executeHttpRequest(MakeAsyncHttpRequestStage.java:151)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$1(MakeAsyncHttpRequestStage.java:104)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
~[?:1.8.0_432]
at
java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683)
~[?:1.8.0_432]
at
java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010)
~[?:1.8.0_432]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:100)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:65)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(AsyncApiCallAttemptMetricCollectionStage.java:62)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(AsyncApiCallAttemptMetricCollectionStage.java:41)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.attemptExecute(AsyncRetryableStage.java:144)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:136)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.execute(AsyncRetryableStage.java:95)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:79)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:44)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:41)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:29)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:64)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:36)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:49)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:32)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:215)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(BaseAsyncClientHandler.java:288)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:227)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:80)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:294)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:73)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:49)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
software.amazon.awssdk.services.cloudwatch.DefaultCloudWatchAsyncClient.putMetricData(DefaultCloudWatchAsyncClient.java:2601)
~[aws-sdk-java-bundle-2.23.18.jar:?]
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:229)
~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
... 6 more
25/04/15 08:30:14 INFO YarnClusterSchedulerBackend: Shutting down all
executors
25/04/15 08:30:14 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each
executor to shut down
```
i tried reducing the report interval and then i got the following error:
```
25/05/16 10:02:37 WARN Metrics: Error while closing reporter
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@abaa245 rejected from
java.util.concurrent.ThreadPoolExecutor@47d4d62[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 2]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_432]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
~[?:1.8.0_432]
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
~[?:1.8.0_432]
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
~[?:1.8.0_432]
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient.putMetricDataAsync(AmazonCloudWatchAsyncClient.java:1374)
~[aws-java-sdk-bundle-1.12.705.jar:?]
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient.putMetricDataAsync(AmazonCloudWatchAsyncClient.java:1366)
~[aws-java-sdk-bundle-1.12.705.jar:?]
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:229)
~[hudi-aws_0.13.1-HudiMetricsReporter-1.0.jar:?]
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212)
~[hudi-aws_0.13.1-HudiMetricsReporter-1.0.jar:?]
at
org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
~[hudi-spark3.5-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at
org.apache.hudi.metrics.cloudwatch.CloudWatchMetricsReporter.report(CloudWatchMetricsReporter.java::55)
~[hudi-aws_0.13.1-HudiMetricsReporter-1.0.jar:?]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_432]
at org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:118)
~[hudi-spark3.5-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_432]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
