[
https://issues.apache.org/jira/browse/HUDI-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangxianghu updated HUDI-1151:
------------------------------
Description:
I get this NPE when using HoodieDeltaStreamer to etl kafka data to hudi table.
the topic is newly created, and there is no data in the topic when the job is
started
{code:java}
20/08/05 21:18:00 INFO helpers.KafkaOffsetGen: SourceLimit not configured, set
numEvents to default value : 5000000
20/08/05 21:18:00 INFO sources.JsonKafkaSource: About to read 0 from Kafka for
topic :hudi_test_callback
20/08/05 21:18:00 INFO deltastreamer.DeltaSync: No new data, source checkpoint
has not changed. Nothing to commit. Old
checkpoint=(Option{val=hudi_test_callback,0:0}). New
Checkpoint=(hudi_test_callback,0:0)
20/08/05 21:18:00 ERROR deltastreamer.HoodieDeltaStreamer: Shutting down
delta-sync due to exception
java.lang.NullPointerException
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
20/08/05 21:18:00 INFO deltastreamer.HoodieDeltaStreamer: Delta Sync shutdown.
Error ?true
20/08/05 21:18:00 ERROR async.AbstractAsyncService: Monitor noticed one or more
threads failed. Requesting graceful shutdown of other threads
java.util.concurrent.ExecutionException:
org.apache.hudi.exception.HoodieException
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at
org.apache.hudi.async.AbstractAsyncService.lambda$monitorThreads$0(AbstractAsyncService.java:136)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:585)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
... 3 more
Caused by: java.lang.NullPointerException
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
... 4 more
20/08/05 21:18:00 ERROR async.AbstractAsyncService: Service shutdown with error
java.util.concurrent.ExecutionException:
org.apache.hudi.exception.HoodieException
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at
org.apache.hudi.async.AbstractAsyncService.waitForShutdown(AbstractAsyncService.java:72)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:152)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:149)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:454)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: org.apache.hudi.exception.HoodieException
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:585)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
... 4 more
{code}
was:
I get this NPE when use
h1. HoodieDeltaStreamer
> Fix NPE when no new data in kafka using HoodieDeltaStreamer
> -----------------------------------------------------------
>
> Key: HUDI-1151
> URL: https://issues.apache.org/jira/browse/HUDI-1151
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
> Reporter: wangxianghu
> Assignee: wangxianghu
> Priority: Major
> Fix For: 0.6.0
>
>
> I get this NPE when using HoodieDeltaStreamer to etl kafka data to hudi
> table. the topic is newly created, and there is no data in the topic when the
> job is started
> {code:java}
> 20/08/05 21:18:00 INFO helpers.KafkaOffsetGen: SourceLimit not configured,
> set numEvents to default value : 5000000
> 20/08/05 21:18:00 INFO sources.JsonKafkaSource: About to read 0 from Kafka
> for topic :hudi_test_callback
> 20/08/05 21:18:00 INFO deltastreamer.DeltaSync: No new data, source
> checkpoint has not changed. Nothing to commit. Old
> checkpoint=(Option{val=hudi_test_callback,0:0}). New
> Checkpoint=(hudi_test_callback,0:0)
> 20/08/05 21:18:00 ERROR deltastreamer.HoodieDeltaStreamer: Shutting down
> delta-sync due to exception
> java.lang.NullPointerException
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 20/08/05 21:18:00 INFO deltastreamer.HoodieDeltaStreamer: Delta Sync
> shutdown. Error ?true
> 20/08/05 21:18:00 ERROR async.AbstractAsyncService: Monitor noticed one or
> more threads failed. Requesting graceful shutdown of other threads
> java.util.concurrent.ExecutionException:
> org.apache.hudi.exception.HoodieException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hudi.async.AbstractAsyncService.lambda$monitorThreads$0(AbstractAsyncService.java:136)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hudi.exception.HoodieException
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:585)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
> ... 4 more
> 20/08/05 21:18:00 ERROR async.AbstractAsyncService: Service shutdown with
> error
> java.util.concurrent.ExecutionException:
> org.apache.hudi.exception.HoodieException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hudi.async.AbstractAsyncService.waitForShutdown(AbstractAsyncService.java:72)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:152)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:149)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:454)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
> Caused by: org.apache.hudi.exception.HoodieException
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:585)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:570)
> ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
