Sure, I've stripped things down to eliminate some variables and I think
I have the problem cornered. I created CSV with a header of "csvA" and
another with "csvB". I'm matching with regex "^csvA$" and "^csvB$"
respectively. You may see what the problem is right way. Nifi doesn't
seem to like '$' as an end of line marker. I've tried both "csvB" and
"^csvB" which work fine and "csvB$" fails, however, this isn't strict
enough for our purposes. We may have cases were one file is "col1, col2"
and another is "col1,col2,col3". This could cause duplicates ingests
later eating resources. Is there a way to mark the end of line in the
regex or am I going to have to do a nested regex filter or something else?
Thanks for the help,
Austin
On 03/16/2017 02:27 PM, James Wing wrote:
Austin,
I think you are on the right track with RouteOnContent. Any chance
you can share a sample CSV header, the settings of your RouteOnContent
processor, including the regex?
Thanks,
James
On Thu, Mar 16, 2017 at 11:14 AM, Austin Heyne <[email protected]
<mailto:[email protected]>> wrote:
Hi,
I have a set of CSV files with headers that utilize various
schemas. I'd like to route the CSV files to processors based on
the schema set in the header. I've tried using the RouteOnContent
processor to sort the files based on "content must contain match"
and a regex statement that matches the first line (header).
However, this is throwing an 'unmatched' on every file I send through.
I've also looked at the ValidateCsv processor but it doesn't
appear that works with the header but rather just validates data
types. Unfortunately this won't work as columns with the same data
type could be in a different order.
Is there a ready made solution for this problem that I missed or
perhaps a more clever way to approach it?
Thanks,
Austin Heyne
