[
https://issues.apache.org/jira/browse/NIFI-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629434#comment-15629434
]
ASF GitHub Bot commented on NIFI-2963:
--------------------------------------
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1166#discussion_r86179969
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/FetchHDFS.java
---
@@ -116,10 +123,40 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
return;
}
+ InputStream stream = null;
+ CompressionCodec codec = null;
+ Configuration conf = getConfiguration();
+ final CompressionCodecFactory compressionCodecFactory = new
CompressionCodecFactory(conf);
+ final CompressionType compressionType =
CompressionType.valueOf(context.getProperty(COMPRESSION_CODEC).toString());
+ final boolean inferCompressionCodec = compressionType ==
CompressionType.AUTOMATIC;
+
+ if (inferCompressionCodec || compressionType !=
CompressionType.NONE) {
+ codec = getCompressionCodec(context, getConfiguration());
+ }
+
final URI uri = path.toUri();
final StopWatch stopWatch = new StopWatch(true);
- try (final FSDataInputStream inStream = hdfs.open(path, 16384)) {
- flowFile = session.importFrom(inStream, flowFile);
+ try {
+
+ final String outputFilename;
+ final String originalFilename = path.getName();
+ stream = hdfs.open(path, 16384);
+
+ // Check if we should infer compression codec
+ if (inferCompressionCodec) {
+ codec = compressionCodecFactory.getCodec(path);
--- End diff --
I am not sure I understand what;s happening here in relation to line:134.
Since both IF statements evaluate ```inferCompressionCodec``` this will always
override the value of 'codec' (the one that was set in line:134) if it's
```true```.
> FetchHDFS should support Compression Codec property
> ---------------------------------------------------
>
> Key: NIFI-2963
> URL: https://issues.apache.org/jira/browse/NIFI-2963
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Pierre Villard
> Assignee: Pierre Villard
> Fix For: 1.2.0
>
>
> GetHDFS offers the possibility to define a compression codec in order to
> uncompress data on the fly while retrieving data from HDFS. FetchHDFS
> processor should offer the same option.
> Workaround : use CompressContent processor after FetchHDFS processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
