As a possible work around, there were date functions added to record path in 1.5.0, so if you had a schema that treated the field as a string, you could reformat the column in place using UpdateRecord to get it into whatever format it needs to be in.
On Tue, Feb 13, 2018 at 9:17 PM Koji Kawamura <ijokaruma...@gmail.com> wrote: > Hi Derek, > > By looking at the code briefly, I guess you are using ValidateRecord > processor with CSVReader and AvroWriter.. > As you pointed out, it seems DataTypeUtils.isCompatibleDataType does > not use the date format user defined at CSVReader. > > Is it possible for you to share followings for us to reproduce and > understand it better? > - Sample input CSV file > - NiFi flow template using CSVReader and AvroWriter > > Thanks, > Koji > > On Wed, Feb 14, 2018 at 7:11 AM, Derek Straka <de...@asterius.io> wrote: > > I have a question about the expected behavior of convertSimpleIfPossible > in > > CSVRecordReader.java (NiFi 1.5.0). > > > > I have a custom CSV file that I am taking to an avro schema using > > ValidateRecord. The schema contains a logical date type and the CSV has > > the date in the format MM/DD/YYYY. I expected to provide the date string > > in the controller element for the CSV reader and have everything parse > > happily, but it ends up throwing an exception when it tries to parse > things > > in the avro writer (String->Date). I don't think I should be blaming the > > avro writer because I expected the CSV reader to parse the date for me. > > > > I did a little digging in the CSVRecordReader.java, and I see everything > > flows through convertSimpleIfPossible when parsing the data, and each > data > > type is checked with DataTypeUtils.isCompatibleDataType prior to actually > > trying to perform the conversion. > > > > The date string doesn't use the user provided format in the call to > > DataTypeUtils.isCompatibleDataType, but instead uses the default for date > > types. The validation ends up failing when it uses the default date > string > > (YYYY-MM-DD), so it won't use LAZY_DATE_FORMAT as I expected. Am I > totally > > off base, or it this unexpected behavior? > > > > Thanks. > > > > -Derek > -- Sent from Gmail Mobile