kbastani opened a new issue #8117:
URL: https://github.com/apache/pinot/issues/8117
When ingesting a `Timestamp` field as a String that has an SDF (Simple Date
Format) pattern that is non-default for `java.sql.Timestamp` in a schema's
dimension field spec (which requires `yyyy-mm-dd hh:mm:ss[.fffffffff]`) Pinot
will fail to import the record. To work around this, the source data must
either be modified to match the standard SDF or transformed in the table spec.
Example:
```json
"dimensionFieldSpecs": [{
"name": "Date",
"dataType": "TIMESTAMP"
}]
```
Source data:
```csv
ID, DATE
1, "10/1/2005",
```
The SDF format here is not specified, since time granularity is not allowed
as a dimension. The same issue can be experienced for ingesting a String field
as a `Timestamp` in the date field specification of a schema, which does allow
a user to specify an SDF. There are some scenarios where using the dimension
field specification to automatically convert from a String in the source
dataset to a `Timestamp` is useful and simpler than using the date time field
spec.
The solution to this issue is to allow for conversions to a `Timestamp` from
a String as an input that automatically infers popular SDFs that already have a
pre-defined granularity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
