[
https://issues.apache.org/jira/browse/HUDI-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004569#comment-17004569
]
Balaji Varadarajan commented on HUDI-474:
-----------------------------------------
[~srkhan] As per your comment in the email thread - "when I have checked,
folder .aux was empty ... ", this means that you have upgraded deltastreamer
to latest master (which includes
[https://github.com/apache/incubator-hudi/pull/1009]) when running earlier
clean operations but from the stack trace, it looks like older layout version
is used.
Is it possible you are using 0.5.0 version of deltastreamer or have you set the
configuration. hoodie.timeline.layout.version. In any case, can you attach the
complete logs of your deltastreamer run (where you saw the problem) in this
Jira ? I did not see the attachment in your email to the dev ML.
Balaji.V
> Delta Streamer is not able to read the commit files
> ---------------------------------------------------
>
> Key: HUDI-474
> URL: https://issues.apache.org/jira/browse/HUDI-474
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: DeltaStreamer
> Reporter: Shahida Khan
> Assignee: Balaji Varadarajan
> Priority: Major
> Fix For: 0.5.1
>
> Attachments: Gmail - Commit time issue in DeltaStreamer
> (Real-Time).pdf
>
>
> DeltaStreamer is not to able to read the correct commit files under when job
> is deployed realtime.
> below is the stack trace:
> ava.util.concurrent.ExecutionException:
> org.apache.hudi.exception.HoodieException: Could not read commit
> details from
> hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at
> org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService.waitForShutdown(AbstractDeltaStreamerService.java:72)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:117)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:297)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
> java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:688)Caused
> by: org.apache.hudi.exception.HoodieException: Could not read commit
> details from
> hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:411)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> java.lang.Thread.run(Thread.java:748)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
