You may have to check the implementation of the customer writer which seems to be com.custom.gobblin.ParquetHDFSDataWriter. It seems to be expecting the file being present which is not there.
On Fri, Nov 10, 2017 at 11:04 AM, Mohan <mohandoss...@gmail.com> wrote: > 2017-11-10 10:57:28 IST ERROR [ForkExecutor-0] gobblin.runtime.Fork 177 - > Fork 0 of task task__1510291622475_0 failed to process data records > java.io.IOException: com.github.rholder.retry.RetryException: Retrying > failed to complete successfully after 5 attempts. > at gobblin.writer.RetryWriter.callWithRetry(RetryWriter.java:133) > at gobblin.writer.RetryWriter.write(RetryWriter.java:113) > at gobblin.runtime.Fork.processRecords(Fork.java:417) > at gobblin.runtime.Fork.run(Fork.java:173) > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.github.rholder.retry.RetryException: Retrying failed to > complete successfully after 5 attempts. > at com.github.rholder.retry.Retryer.call(Retryer.java:174) > at com.github.rholder.retry.Retryer$RetryerCallable.call( > Retryer.java:318) > at gobblin.writer.RetryWriter.callWithRetry(RetryWriter.java:131) > ... 8 more > Caused by: com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Staging output file > /app/gobblin/0.9.0//staging/logs/attempt_local792278526_ > 0001_m_000012_0/log/2017/11/06/part.task__1510291622475_0_4.parquet does > not exist > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) > at com.google.common.cache.LocalCache.get(LocalCache.java:3932) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) > at com.google.common.cache.LocalCache$LocalLoadingCache. > get(LocalCache.java:4806) > at gobblin.writer.PartitionedDataWriter.write(PartitionedDataWriter.java: > 114) > at gobblin.writer.RetryWriter$2.call(RetryWriter.java:108) > at gobblin.writer.RetryWriter$2.call(RetryWriter.java:105) > at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call( > AttemptTimeLimiters.java:78) > at com.github.rholder.retry.Retryer.call(Retryer.java:160) > ... 10 more > Caused by: java.lang.IllegalArgumentException: Staging output file > /app/gobblin/0.9.0//staging/logs/attempt_local792278526_ > 0001_m_000012_0/log/2017/11/06/part.task__1510291622475_0_4.parquet does > not exist > at com.google.common.base.Preconditions.checkArgument( > Preconditions.java:93) > at gobblin.writer.FsDataWriter.setStagingFileGroup(FsDataWriter.java:154) > at com.custom.gobblin.ParquetHDFSDataWriter.<init>( > ParquetHDFSDataWriter.java:61) > at com.custom.gobblin.ParquetDataWriterBuilder.build( > ParquetDataWriterBuilder.java:32) > at gobblin.writer.PartitionedDataWriter.createPartitionWriter( > PartitionedDataWriter.java:186) > at gobblin.writer.PartitionedDataWriter.access$000(PartitionedDataWriter. > java:50) > at gobblin.writer.PartitionedDataWriter$1.load( > PartitionedDataWriter.java:74) > at gobblin.writer.PartitionedDataWriter$1.load( > PartitionedDataWriter.java:69) > at com.google.common.cache.LocalCache$LoadingValueReference. > loadFuture(LocalCache.java:3522) > at com.google.common.cache.LocalCache$Segment.loadSync( > LocalCache.java:2315) > at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache. > java:2278) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) > ... 18 more > 2017-11-10 10:57:28 IST INFO [LocalJobRunner Map Task Executor #0] > gobblin.runtime.GobblinMultiTaskAttempt 91 - All assigned tasks of job > job__1510291622475 have completed in container > attempt_local792278526_0001_m_000012_0 > 2017-11-10 10:57:28 IST INFO [LocalJobRunner Map Task Executor #0] > gobblin.runtime.AbstractJobLauncher 720 - Will commit tasks directly. > 2017-11-10 10:57:28 IST ERROR [Task-committing-pool-0] > gobblin.runtime.Task 229 - Task task__1510291622475_0 failed > gobblin.runtime.ForkException: Fork branches [0] failed for task > task__1510291622475_0 > at gobblin.runtime.Task.commit(Task.java:561) > at gobblin.runtime.GobblinMultiTaskAttempt$1$1. > call(GobblinMultiTaskAttempt.java:114) > at gobblin.runtime.GobblinMultiTaskAttempt$1$1. > call(GobblinMultiTaskAttempt.java:109) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > On Nov 10, 2017 9:43 AM, "Vicky Kak" <vicky....@gmail.com> wrote: > >> Can you attach the complete log file and explain in brief the pull file >> configuration? The above description seems to be a warning and I am not >> sure where it coming from. Also please state the gobblin version you have >> been using as it would become easy to look into the code. >> >> On Fri, Nov 10, 2017 at 6:08 AM, Mohan <mohandoss...@gmail.com> wrote: >> >>> I am getting only this exception in log and in the staging directory the >>> file is not created . Is there any way to run gobblin in debug mode to >>> trace my issue? >>> >>> On Nov 10, 2017 5:43 AM, "Abhishek Tiwari" <a...@apache.org> wrote: >>> >>>> Just check the staging location when the job runs, also check through >>>> the logs if you see any other exceptions. >>>> >>>> On Thu, Nov 9, 2017 at 4:04 PM, Mohan <mohandoss...@gmail.com> wrote: >>>> >>>>> No , how I can check either the file being created or not? >>>>> >>>>> On Nov 9, 2017 11:57 PM, "Shirshanka Das" <shirsha...@apache.org> >>>>> wrote: >>>>> >>>>>> This seems like the cause: >>>>>> "java.lang.IllegalArgumentException: Staging output file >>>>>> /app/gobblin/0.9.0/task-staging/data/attempt_local1524912904 >>>>>> _0001_m_000012_0/log/2017/11/06/part.task_1510232733253_0_0.parquet >>>>>> does not exist" >>>>>> >>>>>> Any idea why the file was not generated? >>>>>> >>>>>> >>>>>> >>>>>> On 2017-11-09 07:22, Mohan <mohandoss...@gmail.com> wrote: >>>>>> > 2017-11-09 18:35:42 IST WARN [ForkExecutor-0] >>>>>> gobblin.writer.RetryWriter$1 >>>>>> > 87 - Caught exception. This may be retried. >>>>>> > com.google.common.util.concurrent.UncheckedExecutionException: >>>>>> > java.lang.IllegalArgumentException: Staging output file >>>>>> > /app/gobblin/0.9.0/task-staging/data/attempt_local1524912904 >>>>>> _0001_m_000012_0/log/2017/11/06/part.task_1510232733253_0_0.parquet >>>>>> > does not exist >>>>>> > at com.google.common.cache.LocalCache$Segment.get(LocalCache.ja >>>>>> va:2199) >>>>>> > at com.google.common.cache.LocalCache.get(LocalCache.java:3932) >>>>>> > at >>>>>> > I'm new to gobblin I'm getting above error. Please any one help me >>>>>> in this >>>>>> > >>>>>> >>>>> >>>> >>
