As a possible work around, there were date functions added to record path
in 1.5.0, so if you had a schema that treated the field as a string, you
could reformat the column in place using UpdateRecord to get it into
whatever format it needs to be in.

On Tue, Feb 13, 2018 at 9:17 PM Koji Kawamura <ijokaruma...@gmail.com>
wrote:

> Hi Derek,
>
> By looking at the code briefly, I guess you are using ValidateRecord
> processor with CSVReader and AvroWriter..
> As you pointed out, it seems DataTypeUtils.isCompatibleDataType does
> not use the date format user defined at CSVReader.
>
> Is it possible for you to share followings for us to reproduce and
> understand it better?
> - Sample input CSV file
> - NiFi flow template using CSVReader and AvroWriter
>
> Thanks,
> Koji
>
> On Wed, Feb 14, 2018 at 7:11 AM, Derek Straka <de...@asterius.io> wrote:
> > I have a question about the expected behavior of convertSimpleIfPossible
> in
> > CSVRecordReader.java (NiFi 1.5.0).
> >
> > I have a custom CSV file that I am taking to an avro schema using
> > ValidateRecord.  The schema contains a logical date type and the CSV has
> > the date in the format MM/DD/YYYY.  I expected to provide the date string
> > in the controller element for the CSV reader and have everything parse
> > happily, but it ends up throwing an exception when it tries to parse
> things
> > in the avro writer (String->Date).  I don't think I should be blaming the
> > avro writer because I expected the CSV reader to parse the date for me.
> >
> > I did a little digging in the CSVRecordReader.java, and I see everything
> > flows through convertSimpleIfPossible when parsing the data, and each
> data
> > type is checked with DataTypeUtils.isCompatibleDataType prior to actually
> > trying to perform the conversion.
> >
> > The date string doesn't use the user provided format in the call to
> > DataTypeUtils.isCompatibleDataType, but instead uses the default for date
> > types.  The validation ends up failing when it uses the default date
> string
> > (YYYY-MM-DD), so it won't use LAZY_DATE_FORMAT as I expected.  Am I
> totally
> > off base, or it this unexpected behavior?
> >
> > Thanks.
> >
> > -Derek
>
-- 
Sent from Gmail Mobile

Reply via email to