I'm using NiFi 1.9.2 QueryDatabaseTable->PutBigQueryBatch to attempt to replicate a MySQL table to BigQuery. In QueryDatabaseTable I've configured 'Use Avro Logical Types=true', so I have a MySQL DATETIME which is encoded in Avro as a Long with logical type timestamp-millis. The PutBigQueryBatch does not support the BigQuery use_avro_logical_types option, so I have an explicit schema where I cast the Long to a BigQuery TIMESTAMP.
The issue, though, is the Long is being interpreted by BQ as a Timestamp with microseconds, and so the resulting Timestamp is off by x1000. Does anyone have a suggestion for a workaround? I've tried an UpdateRecord processor with :multiply(1000), but it has a type conversion exception when writing the new Avro file. -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
