Re: CSV record parsing with custom date formats

2018-02-15 Thread Derek Straka
Koji, Yes. Those are accurate descriptions of the issue. I'm going to start working on NIFI-4882. Hopefully I can have a PR up sometime today. Thanks for your help. -Derek -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: CSV record parsing with custom date formats

2018-02-15 Thread Koji Kawamura
Hi Derek, Thanks for sharing the files and detailed README. I was able to reproduce the issue. It seems there are two different points those can be improved in this scenario. I've created two JIRAs: CSVRecordReader should utilize specified date/time/timestamp format at its

Re: CSV record parsing with custom date formats

2018-02-14 Thread Derek Straka
Thanks for the suggestion. I want to use the field as a date, and my current work around is to preprocess the field with a Jython script. If it is a bug, I'll make a report and take a stab at a fix. -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: CSV record parsing with custom date formats

2018-02-14 Thread Derek Straka
Koji, Thanks for the reply. You are correct on your assumptions. I've created an archive with templates (working and not), sample data, a README, and a hacky patch that fixes the symptom. They can be found here

Re: CSV record parsing with custom date formats

2018-02-13 Thread Bryan Bende
As a possible work around, there were date functions added to record path in 1.5.0, so if you had a schema that treated the field as a string, you could reformat the column in place using UpdateRecord to get it into whatever format it needs to be in. On Tue, Feb 13, 2018 at 9:17 PM Koji Kawamura

Re: CSV record parsing with custom date formats

2018-02-13 Thread Koji Kawamura
Hi Derek, By looking at the code briefly, I guess you are using ValidateRecord processor with CSVReader and AvroWriter.. As you pointed out, it seems DataTypeUtils.isCompatibleDataType does not use the date format user defined at CSVReader. Is it possible for you to share followings for us to