Hi Derek, By looking at the code briefly, I guess you are using ValidateRecord processor with CSVReader and AvroWriter.. As you pointed out, it seems DataTypeUtils.isCompatibleDataType does not use the date format user defined at CSVReader.
Is it possible for you to share followings for us to reproduce and understand it better? - Sample input CSV file - NiFi flow template using CSVReader and AvroWriter Thanks, Koji On Wed, Feb 14, 2018 at 7:11 AM, Derek Straka <de...@asterius.io> wrote: > I have a question about the expected behavior of convertSimpleIfPossible in > CSVRecordReader.java (NiFi 1.5.0). > > I have a custom CSV file that I am taking to an avro schema using > ValidateRecord. The schema contains a logical date type and the CSV has > the date in the format MM/DD/YYYY. I expected to provide the date string > in the controller element for the CSV reader and have everything parse > happily, but it ends up throwing an exception when it tries to parse things > in the avro writer (String->Date). I don't think I should be blaming the > avro writer because I expected the CSV reader to parse the date for me. > > I did a little digging in the CSVRecordReader.java, and I see everything > flows through convertSimpleIfPossible when parsing the data, and each data > type is checked with DataTypeUtils.isCompatibleDataType prior to actually > trying to perform the conversion. > > The date string doesn't use the user provided format in the call to > DataTypeUtils.isCompatibleDataType, but instead uses the default for date > types. The validation ends up failing when it uses the default date string > (YYYY-MM-DD), so it won't use LAZY_DATE_FORMAT as I expected. Am I totally > off base, or it this unexpected behavior? > > Thanks. > > -Derek
