Hi Aruna, If you can not upgrade from NiFi 1.2.0, then I think the best bed is using: ScriptedReader or ScriptedRecordSetWriter for data conversion #1, 2 and 3. As Russ mentioned, EL might be helpful when you implement the scripted components.
#4 is a bit harder since it requires a database connection, but doable, I don't know it works efficiently though.. Alternative approach would be create a temporary table, insert rows as they are, then perform a insert/update query using the temporary table and postgresql lookup table such as: "insert into X select a, b, c, from T inner join L on X.j = T.j where d = ..." Probably data conversion #1, 2, and 3 can be performed from this query as well. Thanks, Koji On Thu, Oct 12, 2017 at 11:50 PM, Russell Bateman <[email protected]> wrote: > Aruna, > > I don't think there is any generalized NiFi training course yet. I started > writing a book a year ago, but it's a pretty thankless task and takes a lot > of time. I mostly write custom processors so I tend to see the world > differently from most "user" users of NiFi. When I answer questions, which > I don't do too much, I often tell an impractical story that doesn't answer > the question asked. > > Now, I haven't done much database work at all, so I'm not going to be too > much help to you. However, looking at your questions, particularly the one > about obtaining the current date when the flow is run (#2) and also #3, I > would suggest that you study the NiFi Expression Language. This would allow > you to insert the "now" date into your flow at some point. > > The other suggestions I have are to experiment, which I'm sure you're > doing, and Google hard for help. For this forum, which is filled with very > nice and helpful people, you'll tend to get a lot more and better help if > you come in with a very specific question rather than a list of things or a > general, "help me" sort of plea. > > Cheers, > > Russ > > P.S. NiFi absolutely rocks, which you'll see as soon as you get over your > initial hump here. But, you're on the right track. > > On 10/12/2017 08:16 AM, Aruna Sankaralingam wrote: > > Hi, > > > > Could you someone please help me with these requirements in the email > below? > > > > Thanks > > Aruna > > > > *From:* Aruna Sankaralingam [mailto:[email protected] > <[email protected]>] > *Sent:* Wednesday, October 11, 2017 11:26 AM > *To:* [email protected] > *Subject:* Transformations using Nifi > > > > I am trying to see what kind of transformations that can be done in nifi > and how. > > Now I have a basic flow that takes CSV from the local dir and puts into s3 > and loads into postgres database. > > There are 4 columns in my test file 3 of which are string and one is an > integer field. I would like to do the following before I load the data into > postgres. If someone can help me on how to go about these, it will be great. > > 1. Convert one of the string columns to upper case > > *For converting to upper case, I was told to use the Update Record > processor but my version is 1.2.0 and the update record processor is not > available. * > > > > 2. Postgres has an extra column called “Load_Date” in which I would > like to load the current date with timestamp when the flow is run > > 3. If the integer column has more than 5 digits, I would like to > take only the first 5 digits and load to the table > > 4. There is a look up table in postgres. I would like to check if > the first column value is present in the look up table and if yes, proceed > ahead and if not ignore the record > > > > I am trying to learn nifi so I would really appreciate any kind of help > here. Is there any training available online that I can take in order to > understand and do all these? > > > > >
