Hi Mark,

Surely, PutFile is not the component to merge the content. I'd plan to use
some other component and you have given me good pointers. Thank you!

Regards,
Valencia



From:   Mark Payne <marka...@hotmail.com>
To:     Valencia Serrao <vser...@us.ibm.com>, "users@nifi.apache.org"
            <users@nifi.apache.org>
Date:   02/21/2018 06:37 PM
Subject:        Re: [Data Flow] File content not read completely



Hey Valencia,

I don't believe that PutFile allows you to append to a file, because doing
so is rife with
problems if you encounter any kind of error (IOException, for example) or
if NiFi restarts
in between. Instead, you should take a look at MergeContent. You can set
the "Merge Strategy"
to "Defragment" in order to re-assemble the FlowFiles that were split apart
via SplitText.

That being said, splitting the data apart, then using ExtractText, and
merging back together can
be quite expensive. If your data is JSON or CSV, then you should probably
look into using the Record-Based
Processors (PublishKafkaRecord, QueryRecord/PartitionRecord). This allows
you to avoid ever splitting
the data apart to begin with and as a result can perform dramatically
better.

Thanks
-Mark


      On Feb 21, 2018, at 7:27 AM, Valencia Serrao <vser...@us.ibm.com>
      wrote:



      Hi Mark,

      Yes! I could get all the required entries with the respective matched
      and unmatched segregated in different folders. Thanks a lot, Mark!!
      My next plan is to check and see how to append all the flowfiles with
      matched entries in one file.

      Regards,
      Valencia

      <graycol.gif>Valencia Serrao---02/16/2018 06:00:47 PM---Hi Mark,
      Thanks for looking into this. I am trying to put in the components
      you have suggested. I'll

      From: Valencia Serrao/Austin/Contr/IBM
      To: marka...@hotmail.com
      Cc: users@nifi.apache.org
      Date: 02/16/2018 06:00 PM
      Subject: Re: [Data Flow] File content not read completely




      Hi Mark,

      Thanks for looking into this. I am trying to put in the components
      you have suggested. I'll update.

      Regards,
      Valencia


      <graycol.gif>Mark Payne ---02/15/2018 07:09:32 PM---Valencia, The
      SplitText processor does not change the ‘filename’ attribute of the
      FlowFile. So you w

      From: Mark Payne <marka...@hotmail.com>
      To: "users@nifi.apache.org" <users@nifi.apache.org>
      Date: 02/15/2018 07:09 PM
      Subject: Re: [Data Flow] File content not read completely



      Valencia,

      The SplitText processor does not change the ‘filename’ attribute of
      the FlowFile. So you will end up with multiple FlowFiles having the
      same name. PutFile may well be overwriting the same file many times -
      or failing to to write the files do to filename conflicts. You can
      resolve this, if it’s your problem, by adding an UpdateAttribute to
      your flow just before PutFile and changing the filename to something
      unique like ${UUID()} or ${filename}.${nextInt()}

      Hope this helps!

      -Mark

      Sent from my iPhone

      On Feb 15, 2018, at 4:59 AM, Valencia Serrao <vser...@us.ibm.com>
      wrote:
                  Hi All,

                  I've started hands-on with Nifi. Basic flows I was able
                  to do without any
                  issues. But currently I've tried adding more steps to the
                  flow.

                  Flow intent: Get a local file, split the text on new
                  line, extract text based on regex, Put matched/unmatched
                  data on respective kafka topics and
                  finally write the kafka contents on the local targets set
                  in PutFile.
                  Current Flow steps: GetFile, SplitText, ExtractText,
                  PutKafka -( 2 of them,one for matched and unmatched), and
                  2 PutFiles components.

                  The issue I'm facing is that - after the flow execution I
                  see only one entry in each of the 2 PutFile targets and
                  rest of the content is not written to them even if the
                  criteria is matched. I feel its not looping through the
                  whole file or something like that. But I had read that
                  Nifi flow is executed for all contents in source files.
                  Maybe I've missed some config somewhere.

                  It would be really helpful if anyone could help on this
                  issue.

                  Regards,
                  Valencia






Reply via email to