To show the problem please find the sample tracelog generated by nifi 1.16.1
Regards,
Mike
From: Michal Tomaszewski <[email protected]>
Date: Wednesday, 4 May 2022 at 10:01
To: [email protected] <[email protected]>
Subject: NiFi 1.16.1 Re: nifi 15.3 - puthive3streaming
Hi,
I think the issue described below was not solved yet. In last version (1.16.1)
the problem still exists.
We used the same compilation options and when starting 1.16.1 log shows
java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost.
Can you send any workaround?
Do we still need to use nar from 1.15.1 version or there is any patch for it?
Regards,
Mike
From: Michal Tomaszewski <[email protected]>
Date: Friday, 21 January 2022 at 15:37
To: [email protected] <[email protected]>
Subject: RE: nifi 15.3 - puthive3streaming
Joe,
Please find trace enclosed.
Martin wrote it works for him with 1.15.1 nar.
Regards,
Mike
-----Original Message-----
From: Bryan Bende <[email protected]>
Sent: Friday, January 21, 2022 3:29 PM
To: [email protected]
Subject: Re: nifi 15.3 - puthive3streaming
This is a known issue starting with the 1.15.0 release.
It was introduced by this change:
https://issues.apache.org/jira/browse/NIFI-9248
The JIRA to correct the problem is:
https://issues.apache.org/jira/browse/NIFI-9392
On Fri, Jan 21, 2022 at 9:17 AM Joe Witt <[email protected]> wrote:
>
> Hello
>
> We need to see the full stack trace. Most likely involves log4j
>
> Thanks
>
> On Fri, Jan 21, 2022 at 7:14 AM Hengesbach, Martin
> <[email protected]> wrote:
>>
>> Hi Mike,
>>
>>
>>
>> I have downloaded the compiled version of nifi-hive3-nar-1.15.3.nar and have
>> the same problem as you. In my NiFi 15.3 installation I have replaced the
>> nifi-hive3-nar-1.15.3.nar through the nifi-hive3-nar-1.15.1.nar and it
>> worked again. So it must be a problem with the 15.3 version and not a
>> compilation problem on your side.
>>
>>
>>
>> Kind regards
>>
>> Martin
>>
>>
>>
>> Von: Michal Tomaszewski [mailto:[email protected]]
>> Gesendet: Freitag, 21. Januar 2022 13:30
>> An: [email protected]
>> Betreff: nifi 15.3 - puthive3streaming
>>
>>
>>
>> Hi again,
>>
>>
>>
>> After compiling 15.3 with hive3 using options:
>>
>> mvn -T C2.0 clean install -DskipTests -Dhive.version=3.1.0
>> -Dhbase.version=2.0.2
>>
>>
>>
>> processor puthive3streaming is not working with error message:
>>
>> „java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost”
>>
>>
>>
>> Isn’t it related to this change:
>>
>> https://github.com/apache/nifi/commit/b7c81d6007852d933961d0067030bc807363e2ee
>>
>> ?
>>
>>
>>
>> What should we change to make puthive3streaming working?
>>
>>
>>
>> Regards,
>>
>> Mike
>>
>> ________________________________
>>
>> FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
>> Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB
>> 101892.
>> Geschäftsführerin: Sabine Brünger-Weilandt.
>> Vorsitzender des Aufsichtsrats: MinR Dr. Hans-Josef Linkens.
>>
>> FIZ Karlsruhe ist zertifiziert mit dem Siegel "audit berufundfamilie".
2022-05-04 10:56:30,463 ERROR [Timer-Driven Process Thread-9]
o.a.n.processors.hive.PutHive3Streaming
PutHive3Streaming[id=325112be-0165-1000-0000-00002a76f8f1] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to java.lang.NoClassDefFoundError:
org/apache/hadoop/tracing/SpanReceiverHost
java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at
org.apache.nifi.util.hive.HiveConfigurator.preload(HiveConfigurator.java:75)
at
org.apache.nifi.processors.hive.PutHive3Streaming.setup(PutHive3Streaming.java:372)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1662)
at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-04 10:56:30,749 ERROR [Timer-Driven Process Thread-4]
o.a.n.processors.hive.PutHive3Streaming
PutHive3Streaming[id=fa6c395f-0164-1000-0000-00005f4bf9b4] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to java.lang.NoClassDefFoundError:
org/apache/hadoop/tracing/SpanReceiverHost
java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at
org.apache.nifi.util.hive.HiveConfigurator.preload(HiveConfigurator.java:75)
at
org.apache.nifi.processors.hive.PutHive3Streaming.setup(PutHive3Streaming.java:372)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1662)
at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-04 10:56:31,553 ERROR [Timer-Driven Process Thread-15]
o.a.n.processors.hive.PutHive3Streaming
PutHive3Streaming[id=37ab3a13-bfea-168b-8d22-21f945f72e81] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to java.lang.NoClassDefFoundError:
org/apache/hadoop/tracing/SpanReceiverHost
java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at
org.apache.nifi.util.hive.HiveConfigurator.preload(HiveConfigurator.java:75)
at
org.apache.nifi.processors.hive.PutHive3Streaming.setup(PutHive3Streaming.java:372)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1662)
at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
