zhangyue19921010 opened a new pull request #5174:
URL: https://github.com/apache/hudi/pull/5174
When users trigger 'commit rollback' for a committed instant, it will throw
followed exception when set rollbackUsingMarkers
true as default.
We may need a common strategy for rollback action either rolling back an
uncompleted instant or a finished commit.
Also users could set rollbackUsingMarkers on demands for better performance
when rolling back an uncompleted instant
```
1643654 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:03 INFO Executor: Adding
file:/private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-97304f5c-e7f7-41b3-a535-c646b6ed02c6/userFiles-1d3d3fa9-f552-4939-b630-f0bc3bad7ed0/jetty-http-9.4.15.v20190215.jar
to class loader
1643851 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 710
bytes result sent to driver
1643863 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0)
in 4441 ms on localhost (executor driver) (1/1)
1643868 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have
all completed, from pool
1643874 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO DAGScheduler: ResultStage 0 (collect at
HoodieSparkEngineContext.java:100) finished in 5.984 s
1643881 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO DAGScheduler: Job 0 finished: collect at
HoodieSparkEngineContext.java:100, took 6.062257 s
1643962 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO HoodieActiveTimeline: Loaded instants upto :
Option{val=[==>20220329175957497__rollback__REQUESTED]}
1643962 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO BaseRollbackPlanActionExecutor: Requesting Rollback with
instant time [==>20220329175957497__rollback__REQUESTED]
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 ERROR SparkMain: Fail to execute commandString
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
org.apache.hudi.exception.HoodieRollbackException: Failed to rollback
/Users/yuezhang/tmp/hudiAfTable/forecast_agg commits 20220329175657735
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:706)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:652)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.hudi.cli.commands.SparkMain.rollback(SparkMain.java:447)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.hudi.cli.commands.SparkMain.main(SparkMain.java:98)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at java.lang.reflect.Method.invoke(Method.java:498)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback
strategy on completed instant:[20220329175657735__commit__COMPLETED]
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.<init>(BaseRollbackActionExecutor.java:93)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.<init>(BaseRollbackActionExecutor.java:73)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor.<init>(CopyOnWriteRollbackActionExecutor.java:48)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.table.HoodieSparkCopyOnWriteTable.rollback(HoodieSparkCopyOnWriteTable.java:345)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
at
org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:689)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
... 15 more
1643974 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO SparkUI: Stopped Spark web UI at http://172.24.10.19:4041
1643984 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
1643993 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO MemoryStore: MemoryStore cleared
1643993 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO BlockManager: BlockManager stopped
1644000 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO BlockManagerMaster: BlockManagerMaster stopped
1644002 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
1644164 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO SparkContext: Successfully stopped SparkContext
1644168 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO ShutdownHookManager: Shutdown hook called
1644168 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO ShutdownHookManager: Deleting directory
/private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-97304f5c-e7f7-41b3-a535-c646b6ed02c6
1644172 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer -
22/03/29 18:00:04 INFO ShutdownHookManager: Deleting directory
/private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-61dd4ecc-a2b7-45a1-831b-468ce36d3143
1644562 [Spring Shell] INFO
org.apache.hudi.common.table.HoodieTableMetaClient - Loading
HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg
1644572 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableConfig
- Loading table properties from
/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/hoodie.properties
1644573 [Spring Shell] INFO
org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of
type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
/Users/yuezhang/tmp/hudiAfTable/forecast_agg
1644573 [Spring Shell] INFO
org.springframework.shell.core.JLineShellComponent - Commit 20220329175657735
failed to roll back
```
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test
coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please
describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
