[
https://issues.apache.org/jira/browse/NIFI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749671#comment-15749671
]
Josh Meyer commented on NIFI-3197:
----------------------------------
Okay, I rebuilt my VM and retried my steps. It worked. This should be able to
be closed.
Again, I did the following:
1. copied hadoop native directory from hadoop cluster to the nifi nodes
2. set java.arg.15=-Djava.library.path=/path/to/hadoop/native/directory/on/nifi
Things worked. I'm not sure what was different, but I rebuilt my VM and it
worked.
> Unable to use Snappy Compression Codec on PutHDFS
> -------------------------------------------------
>
> Key: NIFI-3197
> URL: https://issues.apache.org/jira/browse/NIFI-3197
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.0.0, 1.1.0
> Environment: centos 7.2
> NiFi 1.1.0 or NiFi 1.0.0
> HDP 2.4.3.0
> Reporter: Josh Meyer
> Labels: HDFS, Processor, PutHDFS, Snappy
> Attachments: native-directory.png, nifi-puthdfs-snappy-issue.xml,
> processor-config.png
>
>
> When setting the 'Compression code' to SNAPPY on PutHDFS NiFi is unable to
> compress and push data to HDFS. Attached is the sample workflow that is
> having trouble. Below are the errors for NiFi 1.0.0 and the errors for NiFi
> 1.1.0. Same configuration, but the error is a bit different.
> I have tried setting both LD_LIBRARY_PATH as an environment variable, and
> then I tried setting java.library.path in the bootstrap.conf.
> NiFi 1.1.0 nifi-app.log error message:
> {code}
> 2016-12-14 15:13:56,563 ERROR [Timer-Driven Process Thread-6]
> o.apache.nifi.processors.hadoop.PutHDFS
> PutHDFS[id=fdd52005-0158-1000-ac0f-2a87ed12a1e7] Failed to write to HDFS due
> to java.lang.RuntimeException: native snappy library not available: this
> version of libhadoop was built without snappy support.:
> java.lang.RuntimeException: native snappy library not available: this version
> of libhadoop was built without snappy support.
> 2016-12-14 15:13:56,644 ERROR [Timer-Driven Process Thread-6]
> o.apache.nifi.processors.hadoop.PutHDFS
> java.lang.RuntimeException: native snappy library not available: this version
> of libhadoop was built without snappy support.
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
> ~[hadoop-common-2.7.3.jar:na]
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
> ~[hadoop-common-2.7.3.jar:na]
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
> ~[hadoop-common-2.7.3.jar:na]
> at
> org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)
> ~[hadoop-common-2.7.3.jar:na]
> at
> org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:100)
> ~[hadoop-common-2.7.3.jar:na]
> at
> org.apache.nifi.processors.hadoop.PutHDFS$1$1.process(PutHDFS.java:313)
> ~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053)
> ~[na:na]
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:300)
> ~[nifi-hdfs-processors-1.1.0.jar:1.1.0]
> at java.security.AccessController.doPrivileged(Native Method)
> [na:1.8.0_111-debug]
> at javax.security.auth.Subject.doAs(Subject.java:360)
> [na:1.8.0_111-debug]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> [hadoop-common-2.7.3.jar:na]
> at
> org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230)
> [nifi-hdfs-processors-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111-debug]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_111-debug]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_111-debug]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_111-debug]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111-debug]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111-debug]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-debug]
> {code}
> NiFi 1.0.0 nifi-bootstrap.log
> {code}
> 2016-12-14 15:32:50,373 INFO [NiFi Bootstrap Command Listener]
> org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for
> Bootstrap requests on port 40280
> 2016-12-14 15:43:47,752 INFO [NiFi logging handler] org.apache.nifi.StdOut
> FATAL ERROR in native method: JNI received a class argument that is not a
> class
> 2016-12-14 15:43:47,758 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
> Method)
> 2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:232)
> 2016-12-14 15:43:47,759 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
> 2016-12-14 15:43:47,760 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
> 2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:106)
> 2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at org.apache.nifi.processors.hadoop.PutHDFS$1.process(PutHDFS.java:295)
> 2016-12-14 15:43:47,761 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880)
> 2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
> 2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:276)
> 2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> 2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
> 2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> 2016-12-14 15:43:47,762 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> 2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> 2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 2016-12-14 15:43:47,763 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
> at java.lang.Thread.run(Thread.java:745)
> 2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
> Current thread is 139782124459776
> 2016-12-14 15:43:47,764 INFO [NiFi logging handler] org.apache.nifi.StdOut
> Dumping core ...
> 2016-12-14 15:43:48,525 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache
> NiFi appears to have died. Restarting...
> 2016-12-14 15:43:48,545 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Successfully started Apache NiFi with PID 27342
> 2016-12-14 15:43:52,874 INFO [NiFi Bootstrap Command Listener]
> org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for
> Bootstrap requests on port 43961
> 2016-12-14 15:44:09,020 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2016-12-14 15:44:09,053 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2016-12-14 15:44:09,054 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2016-12-14 15:44:09,231 INFO [main] org.apache.nifi.bootstrap.Command Apache
> NiFi has accepted the Shutdown Command and is shutting down now
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
