Hi,
Sorry to piggy back on this thread, but I have pretty much the same issue – I
am splitting log files -> routeoncontent (various paths) two of these paths
(including unmatched), basically need to just get farmed off into a directory
just in case they are needed later.
These go into a MergeContent processor where I would like to merge into one
file – each flowfile content as a line in the file delimited by line feed (as
like the original file), whichever way I try this though doesn’t quite do what
I want. If I try BinaryConcatenation the file ends up as one long line, if TAR
each Flowfile is a separate file in a TAR (not unsurprisingly). There doesn’t
seem to be anyway of merging flow file content into one file (that ideally has
similar functions to be able to compress, specify number of files etc.)
Another related question to the answer below (really helped me out with same
issue), however if I rename the filename early on in my process flow, it
appears to be changed back to its original at MergeContent processor time so I
have to put another UpdateAttributes step in after the Merge to rename the
filename.
The flow is
UpdateAttributes -> RouteOnContent -> UpdateAttribute -> MergeContent -> PutFile
^ ^ ^ ^
| | | |
Filename changed same same reverted
If I put an extra UpdateAttribute before PutFile then fine. Logging at each of
the above points shows filename updated to ${uuid}-${filename}, but at reverted
is back at filename.
Any suggestions on particularly the first question??
Thanks
Conrad
From: Jeff Lord <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Friday, 19 February 2016 at 03:22
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: splitText output appears to be getting dropped
Matt,
Thanks a bunch!
That did the trick.
Is there a better way to handle this out of curiosity? Than writing out a
single line into multiple files.
Each file contains a single string that will be used to build a url.
-Jeff
On Thu, Feb 18, 2016 at 6:00 PM, Matthew Clarke
<[email protected]<mailto:[email protected]>> wrote:
Jeff,
It appears you files are being dropped because your are auto-terminating
the failure relationship on your putFile processor. When the splitText
processor splits the file by lines every new file has the same filename as the
original it came from. My guess is the first file is being worked to disk and
all others are failing because a file of the same name already exists in target
dir. Try adding an UpdateAttribute processor after the splitText to rename all
the files. Easiest way is to append the files uuid to its filename. I also do
not recommend auto-terminating failure relationships except in rare cases.
Matt
On Feb 18, 2016 8:36 PM, "Jeff Lord"
<[email protected]<mailto:[email protected]>> wrote:
I have a pretty simple flow where I query for a list of ids using
executeProcess and than pass that list along to splitText where I am trying to
split on each line to than dynamically build a url further down the line using
updateAttribute and so on.
executeProcess -> splitText -> putFile
For some reason I am only getting one file written with one line.
I would expect something more like 100 files each with one line.
Using the provenance reporter it appears that some of my items are being
dropped.
Time02/18/2016 17:13:46.145 PST
Event DurationNo value set
Lineage Duration00:00:12.187
TypeDROP
FlowFile Uuid7fa42367-490d-4b54-a32f-d062a885474a
File Size14 bytes
Component Id3b37a828-ba2c-4047-ba7a-578fd0684ce6
Component NamePutFile
Component TypePutFile
DetailsAuto-Terminated by failure Relationship
Any ideas on what I need to change here?
Thanks in advance,
Jeff
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report
this email as spam.
SecureData, combating cyber threats
______________________________________________________________________
The information contained in this message or any of its attachments may be
privileged and confidential and intended for the exclusive use of the intended
recipient. If you are not the intended recipient any disclosure, reproduction,
distribution or other dissemination or use of this communications is strictly
prohibited. The views expressed in this email are those of the individual and
not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if
followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered
Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent,
ME16 9NT
