Hi Mark, I am confused about the whole process. I have the following questions:
1. From what I read, I can use TailFile to read the log file. However, it would only read the file once (as the input file does not change). Is there a way that I can read the file every time it gets started? 2. As you suggested, I am writing a personal Python script to handle the count. Most of the examples online are in Jython.(My NiFi version is 1.3.0. I suppose it's similar to Python, correct?) I find https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html <https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html> as a useful guide, but I don't understand what to choose from. What's the input and output for the script? I want to read the file line by line and count the string occurrences. I'm currently using key,value = flowFile.getAttributes().iteritems() to get the file content, but it shows "too many values to unpack". (The original file is 41.14MB) For the output, the common way seems to be using a callback. Is it necessary in my case? Or can I just add these attributes to the output file and extract the attributes later? 3. To write the columns into the SQL table, the common way seems to use "ReplaceText" and "PutSQL". I also noticed there's a processor called PutDatabaseRecord that might combine the function of ExecuteScript and PutSQL together. Since my Python script doesn't work so far I can't really test the result. But is this an easier approach? Any help is appreciated... Thanks, Tina -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
