ksnijjer opened a new issue #7607:
URL: https://github.com/apache/pinot/issues/7607
Currently it seems like Pinot is very rigid about **only** allowing schema
changes where a new column has been added. If you need to make any changes to
existing columns for e.g changing column from int to long or timestamp format
change, which don't affect existing data you are still forced to delete and
recreate the table/schema which is very limiting and depending on amount of
data that will need to be reprocessed perhaps not even feasible.
Here is a sample scenario, I created a schema where one of the datetimeSpec
columns has a ingestion transform
`{
"name": "CreationTimeMillis",
"dataType": "LONG",
"defaultNullValue": 1592703025,
"transformFunction": "fromDateTime(CreateTime,
\"yyyy-MM-dd'T'HH:mm:ssZ\")",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}`
now apparently time format specification here is incorrect and I need to
change this to
`{
"name": "CreationTimeInMillis",
"dataType": "LONG",
"defaultNullValue": 1592703058,
"transformFunction": "fromDateTime(CreateTime,
'yyyy-MM-dd''T''HH:mm:ssZ')",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}`
but this is now considered as a backward incompatible change.
Can we add some additional intelligence in the logic so such changes which
don't affect underlying data are allowed to go through ?
cc @mayankshriv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
