voonhous commented on code in PR #6566:
URL: https://github.com/apache/hudi/pull/6566#discussion_r963232721
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/HoodieFlinkClusteringJob.java:
##########
@@ -335,5 +391,17 @@ public void shutdownAsyncService(boolean error) {
public void shutDown() {
shutdownAsyncService(false);
}
+
+ /**
+ * Execute a dummy pipeline to prevent "no execute() calls" exceptions
from being thrown if
+ * clustering is not performed.
+ */
Review Comment:
While I have yet to test it on service mode, from my understanding of the
code:
Given that an error is already thrown here (non service mode):
```java
try {
compactionScheduleService.compact();
} catch (Exception e) {
LOG.error("Got error running delta sync once. Shutting down", e);
throw e;
} finally {
LOG.info("Shut down hoodie flink compactor");
}
```
The error will be thrown as a `HoodieException` in the same service mode
execution block.
```java
try {
compact();
Thread.sleep(cfg.minCompactionIntervalSeconds * 1000);
} catch (Exception e) {
LOG.error("Shutting down compaction service due to exception", e);
error = true;
throw new HoodieException(e.getMessage(), e);
}
```
The error logs in text form in non-service mode execution:
```java
2022-09-06 11:59:45,633 INFO
org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading
HoodieTableMetaClient from hdfs://hudi_mor_compaction_test/
2022-09-06 11:59:45,638 INFO org.apache.hudi.common.table.HoodieTableConfig
[] - Loading table properties from
hdfs://hudi_mor_compaction_test/.hoodie/hoodie.properties
2022-09-06 11:59:45,643 INFO
org.apache.hudi.common.table.HoodieTableMetaClient [] - Finished
Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
hdfs://hudi_mor_compaction_test/
2022-09-06 11:59:45,643 INFO
org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading
Active commit timeline for hdfs://hudi_mor_compaction_test/
2022-09-06 11:59:46,012 INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Loaded
instants upto : Option{val=[==>20220906115739354__deltacommit__INFLIGHT]}
2022-09-06 11:59:46,018 INFO
org.apache.hudi.common.table.view.FileSystemViewManager [] - Creating View
Manager with storage type :REMOTE_FIRST
2022-09-06 11:59:46,018 INFO
org.apache.hudi.common.table.view.FileSystemViewManager [] - Creating
remote first table view
2022-09-06 11:59:46,020 INFO
org.apache.hudi.sink.compact.HoodieFlinkCompactor [] - Hoodie Flink
Compactor running only single round
2022-09-06 11:59:46,361 INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Loaded
instants upto : Option{val=[==>20220906115739354__deltacommit__INFLIGHT]}
2022-09-06 11:59:46,363 INFO org.apache.hudi.client.RunsTableService
[] - No compaction plan scheduled, turns on the compaction plan
schedule with --schedule option
2022-09-06 11:59:46,363 INFO
org.apache.hudi.sink.compact.HoodieFlinkCompactor [] - Shut down
hoodie flink compactor
2022-09-06 11:59:46,420 ERROR
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap
[] - Application failed unexpectedly. Killing cluster...
java.util.concurrent.CompletionException:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
The application contains no execute() calls.
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
~[?:1.8.0_232]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
~[?:1.8.0_232]
at
java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
~[?:1.8.0_232]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
~[?:1.8.0_232]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
~[?:1.8.0_232]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
~[?:1.8.0_232]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:263)
~[flink-dist_2.11-1.13.17.jar:1.13.17]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$1(ApplicationDispatcherBootstrap.java:219)
~[flink-dist_2.11-1.13.17.jar:1.13.17]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_232]
at
org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:159)
[flink-dist_2.11-1.13.17.jar:1.13.17]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
[flink-dist_2.11-1.13.17.jar:1.13.17]
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
[flink-dist_2.11-1.13.17.jar:1.13.17]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.13.17.jar:1.13.17]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.13.17.jar:1.13.17]
at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.13.17.jar:1.13.17]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.13.17.jar:1.13.17]
Caused by:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
The application contains no execute() calls.
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:257)
~[flink-dist_2.11-1.13.17.jar:1.13.17]
... 10 more
2022-09-06 11:59:46,427 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting
YarnApplicationClusterEntryPoint down with application status FAILED.
Diagnostics null.
2022-09-06 11:59:46,429 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting
down rest endpoint.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
