tooptoop4 opened a new issue #1738:
URL: https://github.com/apache/hudi/issues/1738
```
2020-06-15 20:02:11,059 [main] INFO
com.uber.hoodie.common.table.log.HoodieLogFormat$WriterBuilder - Building
HoodieLogFormat Writer
2020-06-15 20:02:11,060 [main] INFO
com.uber.hoodie.common.table.log.HoodieLogFormat$WriterBuilder - Computing the
next log version for commits in s3a://REDACT/.hoodie/archived
Exception in thread "main" com.uber.hoodie.exception.HoodieException: Unable
to initialize HoodieLogFormat writer
at
com.uber.hoodie.io.HoodieCommitArchiveLog.openWriter(HoodieCommitArchiveLog.java:93)
at
com.uber.hoodie.io.HoodieCommitArchiveLog.archiveIfRequired(HoodieCommitArchiveLog.java:115)
at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:542)
at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:489)
at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:480)
...
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:890)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.FileNotFoundException: No such file or directory:
s3a://REDACT/.hoodie/archived
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1616)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1377)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1353)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)
at com.uber.hoodie.common.util.FSUtils.getAllLogFiles(FSUtils.java:302)
at
com.uber.hoodie.common.util.FSUtils.getLatestLogVersion(FSUtils.java:316)
at
com.uber.hoodie.common.util.FSUtils.getCurrentLogVersion(FSUtils.java:327)
at
com.uber.hoodie.common.table.log.HoodieLogFormat$WriterBuilder.build(HoodieLogFormat.java:184)
at
com.uber.hoodie.io.HoodieCommitArchiveLog.openWriter(HoodieCommitArchiveLog.java:88)
... 27 more
2020-06-15 20:02:11,098 [pool-1-thread-1] INFO
org.apache.spark.SparkContext - Invoking stop() from shutdown hook
current s3 listing:
bash-4.2$ ~/bin/aws s3 ls s3://REDACT/ --recursive
2020-06-01 17:15:15 0 PREREDACT/.hoodie/.aux/
2020-06-01 17:20:13 0 PREREDACT/.hoodie/.temp/
2020-06-01 16:35:18 1091 PREREDACT/.hoodie/20200529234618.clean
2020-06-01 16:35:18 2022 PREREDACT/.hoodie/20200529234618.commit
2020-06-01 16:35:18 1091 PREREDACT/.hoodie/20200529234910.clean
2020-06-01 16:35:18 6150 PREREDACT/.hoodie/20200529234910.commit
2020-06-01 16:35:18 1091 PREREDACT/.hoodie/20200529235222.clean
2020-06-01 16:35:18 3846 PREREDACT/.hoodie/20200529235222.commit
2020-06-01 16:35:18 1091 PREREDACT/.hoodie/20200529235443.clean
2020-06-01 16:35:18 3851 PREREDACT/.hoodie/20200529235443.commit
2020-06-01 16:35:18 1091 PREREDACT/.hoodie/20200529235719.clean
2020-06-01 16:35:18 3850 PREREDACT/.hoodie/20200529235719.commit
2020-06-01 16:35:18 1091 PREREDACT/.hoodie/20200529235933.clean
2020-06-01 16:35:18 3853 PREREDACT/.hoodie/20200529235933.commit
2020-06-01 16:35:18 1091 PREREDACT/.hoodie/20200530000232.clean
2020-06-01 16:35:18 3850 PREREDACT/.hoodie/20200530000232.commit
2020-06-01 16:35:19 1091 PREREDACT/.hoodie/20200530000535.clean
2020-06-01 16:35:19 3854 PREREDACT/.hoodie/20200530000535.commit
2020-06-01 16:35:19 1091 PREREDACT/.hoodie/20200530000831.clean
2020-06-01 16:35:19 3851 PREREDACT/.hoodie/20200530000831.commit
2020-06-01 16:35:19 1091 PREREDACT/.hoodie/20200530001136.clean
2020-06-01 16:35:19 3854 PREREDACT/.hoodie/20200530001136.commit
2020-06-01 16:35:19 1104 PREREDACT/.hoodie/20200530001432.clean
2020-06-01 16:35:19 3851 PREREDACT/.hoodie/20200530001432.commit
2020-06-01 16:35:19 1710 PREREDACT/.hoodie/20200530001729.clean
2020-06-01 16:35:19 3856 PREREDACT/.hoodie/20200530001729.commit
2020-06-04 14:22:24 1104 PREREDACT/.hoodie/20200604132158.clean
2020-06-04 14:22:21 3854 PREREDACT/.hoodie/20200604132158.commit
2020-06-04 14:23:47 1104 PREREDACT/.hoodie/20200604132316.clean
2020-06-04 14:23:43 6205 PREREDACT/.hoodie/20200604132316.commit
2020-06-04 15:31:36 1104 PREREDACT/.hoodie/20200604143105.clean
2020-06-04 15:31:33 3877 PREREDACT/.hoodie/20200604143105.commit
2020-06-04 15:33:02 1104 PREREDACT/.hoodie/20200604143235.clean
2020-06-04 15:32:57 3875 PREREDACT/.hoodie/20200604143235.commit
2020-06-04 15:34:31 1104 PREREDACT/.hoodie/20200604143400.clean
2020-06-04 15:34:27 3875 PREREDACT/.hoodie/20200604143400.commit
2020-06-07 03:05:37 1104 PREREDACT/.hoodie/20200607020507.clean
2020-06-07 03:05:34 3879 PREREDACT/.hoodie/20200607020507.commit
2020-06-07 03:07:17 1104 PREREDACT/.hoodie/20200607020642.clean
2020-06-07 03:07:14 3877 PREREDACT/.hoodie/20200607020642.commit
2020-06-07 03:08:47 1104 PREREDACT/.hoodie/20200607020820.clean
2020-06-07 03:08:44 3877 PREREDACT/.hoodie/20200607020820.commit
2020-06-07 03:12:13 1104 PREREDACT/.hoodie/20200607021138.clean
2020-06-07 03:12:11 6203 PREREDACT/.hoodie/20200607021138.commit
2020-06-07 03:13:36 1104 PREREDACT/.hoodie/20200607021313.clean
2020-06-07 03:13:33 3874 PREREDACT/.hoodie/20200607021313.commit
2020-06-07 03:15:10 1740 PREREDACT/.hoodie/20200607021437.clean
2020-06-07 03:15:07 3879 PREREDACT/.hoodie/20200607021437.commit
2020-06-07 03:16:39 3639 PREREDACT/.hoodie/20200607021611.clean
2020-06-07 03:16:36 3877 PREREDACT/.hoodie/20200607021611.commit
2020-06-07 03:18:13 1740 PREREDACT/.hoodie/20200607021742.clean
2020-06-07 03:18:10 3879 PREREDACT/.hoodie/20200607021742.commit
2020-06-07 03:19:40 1740 PREREDACT/.hoodie/20200607021914.clean
2020-06-07 03:19:38 3876 PREREDACT/.hoodie/20200607021914.commit
2020-06-07 03:21:13 1740 PREREDACT/.hoodie/20200607022043.clean
2020-06-07 03:21:11 3879 PREREDACT/.hoodie/20200607022043.commit
2020-06-07 03:22:52 1740 PREREDACT/.hoodie/20200607022224.clean
2020-06-07 03:22:50 3879 PREREDACT/.hoodie/20200607022224.commit
2020-06-07 03:24:18 1740 PREREDACT/.hoodie/20200607022350.clean
2020-06-07 03:24:15 3876 PREREDACT/.hoodie/20200607022350.commit
2020-06-09 17:04:10 1740 PREREDACT/.hoodie/20200609160339.clean
2020-06-09 17:04:07 6199 PREREDACT/.hoodie/20200609160339.commit
2020-06-09 17:05:35 3874 PREREDACT/.hoodie/20200609160515.commit
2020-06-09 17:07:17 3879 PREREDACT/.hoodie/20200609160644.commit
2020-06-09 17:08:45 3877 PREREDACT/.hoodie/20200609160822.commit
2020-06-09 17:10:18 3881 PREREDACT/.hoodie/20200609160950.commit
2020-06-09 17:11:49 3878 PREREDACT/.hoodie/20200609161125.commit
2020-06-09 17:13:24 3881 PREREDACT/.hoodie/20200609161254.commit
2020-06-09 17:15:12 3881 PREREDACT/.hoodie/20200609161444.commit
2020-06-09 17:16:42 3878 PREREDACT/.hoodie/20200609161618.commit
2020-06-11 10:15:10 6201 PREREDACT/.hoodie/20200611091443.commit
2020-06-11 10:16:51 3876 PREREDACT/.hoodie/20200611091630.commit
2020-06-11 10:18:26 3881 PREREDACT/.hoodie/20200611091758.commit
2020-06-11 10:20:06 3879 PREREDACT/.hoodie/20200611091936.commit
2020-06-11 10:21:43 3881 PREREDACT/.hoodie/20200611092117.commit
2020-06-11 10:23:38 3879 PREREDACT/.hoodie/20200611092249.commit
2020-06-11 10:25:37 3881 PREREDACT/.hoodie/20200611092452.commit
2020-06-11 10:29:37 3881 PREREDACT/.hoodie/20200611092814.commit
2020-06-11 10:31:25 3878 PREREDACT/.hoodie/20200611093049.commit
2020-06-15 21:02:11 3878 PREREDACT/.hoodie/20200615200144.commit
2020-06-15 21:03:42 3881 PREREDACT/.hoodie/20200615200312.commit
2020-06-15 21:05:22 3881 PREREDACT/.hoodie/20200615200453.commit
2020-06-15 21:06:45 3878 PREREDACT/.hoodie/20200615200620.commit
2020-06-15 21:08:14 3881 PREREDACT/.hoodie/20200615200745.commit
2020-06-15 21:09:39 3879 PREREDACT/.hoodie/20200615200914.commit
2020-06-01 17:24:32 194 PREREDACT/.hoodie/hoodie.properties
2020-06-01 16:35:19 93 PREREDACT/unknown/.hoodie_partition_metadata
2020-06-07 03:13:23 687069
PREREDACT/unknown/1ed20eac-618f-4b2b-ab09-9a8f620d52f7_0_20200607021313.parquet
2020-06-09 17:05:26 687126
PREREDACT/unknown/1ed20eac-618f-4b2b-ab09-9a8f620d52f7_0_20200609160515.parquet
2020-06-11 10:16:40 687164
PREREDACT/unknown/1ed20eac-618f-4b2b-ab09-9a8f620d52f7_0_20200611091630.parquet
2020-06-04 14:23:32 686896
PREREDACT/unknown/1ed20eac-618f-4b2b-ab09-9a8f620d52f7_2_20200604132316.parquet
2020-06-07 03:11:56 687025
PREREDACT/unknown/1ed20eac-618f-4b2b-ab09-9a8f620d52f7_2_20200607021138.parquet
2020-06-09 17:03:52 687125
PREREDACT/unknown/1ed20eac-618f-4b2b-ab09-9a8f620d52f7_2_20200609160339.parquet
2020-06-11 10:14:58 687163
PREREDACT/unknown/1ed20eac-618f-4b2b-ab09-9a8f620d52f7_2_20200611091443.parquet
2020-06-04 14:23:30 661660
PREREDACT/unknown/3a7e1b07-4d43-4cef-bad4-fb0b1945452b_0_20200604132316.parquet
2020-06-07 03:11:56 661763
PREREDACT/unknown/3a7e1b07-4d43-4cef-bad4-fb0b1945452b_0_20200607021138.parquet
2020-06-09 17:03:53 661807
PREREDACT/unknown/3a7e1b07-4d43-4cef-bad4-fb0b1945452b_1_20200609160339.parquet
2020-06-11 10:14:54 661856
PREREDACT/unknown/3a7e1b07-4d43-4cef-bad4-fb0b1945452b_1_20200611091443.parquet
2020-06-07 03:08:36 8481028
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607020820.parquet
2020-06-07 03:14:56 8730524
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607021437.parquet
2020-06-07 03:16:24 8731305
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607021611.parquet
2020-06-07 03:18:01 8926445
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607021742.parquet
2020-06-07 03:19:26 8926645
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607021914.parquet
2020-06-07 03:21:02 9137267
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607022043.parquet
2020-06-07 03:22:43 9179806
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607022224.parquet
2020-06-07 03:24:02 9179883
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200607022350.parquet
2020-06-09 17:07:08 9180937
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200609160644.parquet
2020-06-09 17:08:37 9180889
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200609160822.parquet
2020-06-09 17:10:10 9182202
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200609160950.parquet
2020-06-09 17:11:37 9182209
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200609161125.parquet
2020-06-09 17:13:16 9183427
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200609161254.parquet
2020-06-09 17:15:03 9183651
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200609161444.parquet
2020-06-09 17:16:30 9183647
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200609161618.parquet
2020-06-11 10:18:19 9184774
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200611091758.parquet
2020-06-11 10:19:55 9184729
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200611091936.parquet
2020-06-11 10:21:35 9185833
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200611092117.parquet
2020-06-11 10:23:16 9185826
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200611092249.parquet
2020-06-11 10:25:19 9186894
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200611092452.parquet
2020-06-11 10:29:20 9187050
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200611092814.parquet
2020-06-11 10:31:03 9187045
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200611093049.parquet
2020-06-15 21:01:56 9187054
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200615200144.parquet
2020-06-15 21:03:34 9187292
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200615200312.parquet
2020-06-15 21:05:15 9189647
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200615200453.parquet
2020-06-15 21:06:32 9189649
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200615200620.parquet
2020-06-15 21:08:05 9191392
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200615200745.parquet
2020-06-15 21:09:30 9191397
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_0_20200615200914.parquet
2020-06-07 03:11:59 8481102
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_3_20200607021138.parquet
2020-06-09 17:03:58 9179924
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_3_20200609160339.parquet
2020-06-11 10:14:59 9183710
PREREDACT/unknown/7ed927a4-15bf-4e9f-a8a5-04f685a9004b_3_20200611091443.parquet
2020-06-09 17:03:52 657453
PREREDACT/unknown/ba3684e1-3ce8-4b5f-a077-2b8ba8e62787_0_20200609160339.parquet
2020-06-11 10:14:58 657515
PREREDACT/unknown/ba3684e1-3ce8-4b5f-a077-2b8ba8e62787_0_20200611091443.parquet
2020-06-04 14:23:30 657285
PREREDACT/unknown/ba3684e1-3ce8-4b5f-a077-2b8ba8e62787_1_20200604132316.parquet
2020-06-07 03:11:55 657385
PREREDACT/unknown/ba3684e1-3ce8-4b5f-a077-2b8ba8e62787_1_20200607021138.parquet
2020-06-01 13:05:46 0 PREREDACT/unknown/x
5 times retrying the job but always hit the same error.
Using spark 2.3.4 and hoodie 0.4.6, yes i know old but code path looks to be
same:
https://github.com/apache/hudi/blob/hoodie-0.4.6/hoodie-client/src/main/java/com/uber/hoodie/io/HoodieCommitArchiveLog.java
vs
https://github.com/apache/hudi/blob/master/hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
https://github.com/apache/hudi/blob/hoodie-0.4.6/hoodie-common/src/main/java/com/uber/hoodie/common/util/FSUtils.java
vs
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
