As a possible work around, there were date functions added to record path
in 1.5.0, so if you had a schema that treated the field as a string, you
could reformat the column in place using UpdateRecord to get it into
whatever format it needs to be in.
On Tue, Feb 13, 2018 at 9:17 PM Koji Kawamura <ijokaruma...@gmail.com>
> Hi Derek,
> By looking at the code briefly, I guess you are using ValidateRecord
> processor with CSVReader and AvroWriter..
> As you pointed out, it seems DataTypeUtils.isCompatibleDataType does
> not use the date format user defined at CSVReader.
> Is it possible for you to share followings for us to reproduce and
> understand it better?
> - Sample input CSV file
> - NiFi flow template using CSVReader and AvroWriter
> On Wed, Feb 14, 2018 at 7:11 AM, Derek Straka <de...@asterius.io> wrote:
> > I have a question about the expected behavior of convertSimpleIfPossible
> > CSVRecordReader.java (NiFi 1.5.0).
> > I have a custom CSV file that I am taking to an avro schema using
> > ValidateRecord. The schema contains a logical date type and the CSV has
> > the date in the format MM/DD/YYYY. I expected to provide the date string
> > in the controller element for the CSV reader and have everything parse
> > happily, but it ends up throwing an exception when it tries to parse
> > in the avro writer (String->Date). I don't think I should be blaming the
> > avro writer because I expected the CSV reader to parse the date for me.
> > I did a little digging in the CSVRecordReader.java, and I see everything
> > flows through convertSimpleIfPossible when parsing the data, and each
> > type is checked with DataTypeUtils.isCompatibleDataType prior to actually
> > trying to perform the conversion.
> > The date string doesn't use the user provided format in the call to
> > DataTypeUtils.isCompatibleDataType, but instead uses the default for date
> > types. The validation ends up failing when it uses the default date
> > (YYYY-MM-DD), so it won't use LAZY_DATE_FORMAT as I expected. Am I
> > off base, or it this unexpected behavior?
> > Thanks.
> > -Derek
Sent from Gmail Mobile