Joseph Percivall created NIFI-1081:
--------------------------------------
Summary: Add option to ExecuteStreamCommand to put value of
execution to an attribute
Key: NIFI-1081
URL: https://issues.apache.org/jira/browse/NIFI-1081
Project: Apache NiFi
Issue Type: Test
Reporter: Joseph Percivall
Assignee: Joseph Percivall
Priority: Minor
This issue arose from a user on the mailing list. It demonstrates the need to
be able to put the output of ExecuteStreamCommand to an attribute:
I'm looking to process many files into common formats. The source files are
coming in various character sets, mime types, and new line terminators.
My thinking for a data flow was along these lines:
GetFile (from many sub directories) ->
ExecuteStreamCommand (file -i) ->
ConvertCharacterSet (from previous command to utf8) ->
ReplaceText (to change any \r\n into \n) ->
PutFile (into a directory structure based on values found in the original file
path and filename)
Additional steps would be added for archiving a copy of the original,
converting xml files, etc.
Attempting to process these with Nifi leaves me confused as to how to process
within the tool. If I want to ConvertCharacterSet, I have to know the input
type. I setup a ExecuteStreamCommand to file -i
${absolute.path:append(${filename})} which returned the expected values. I
don't see a way to turn these results into input for the processor, which
doesn't accept expression language for that field.
I also considered ConvertCSVToAvro as an interim step but notice the same
issue. Any suggestions what this dataflow should look like?
Bryan Bende's response:
One problem with the above flow is that ExecuteStreamCommand will replace the
contents of the FlowFile with the results of the command, so the FlowFIle will
have the encoding value and no longer have the original content.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
