cbalci opened a new pull request, #14556:
URL: https://github.com/apache/pinot/pull/14556
Adding proper time column support to the Spark writer to correctly create
time information in segment metadata, which was missing before.
Full api for configuring the time column when writing out segments is as
follows:
```scala
dataframe.write.format("pinot")
.mode("append")
.option("table", "airlineStats")
.option("tableType", "OFFLINE")
.option("segmentNameFormat",
"{table}_{startTime}_{endTime}_{partitionId:03}")
.option("timeColumnName", "ts") # <- column
.option("timeFormat", "EPOCH|SECONDS") # <- format
.option("timeGranularity", "1:SECONDS") # <- granularity
.save("myPath")
```
Which follows Pinot's time column config options `timeFormat` and
`timeGranularity`.
I also added a couple new template variables, `startTime` and `endTime`
which can be used for building segment name. This is useful for efficient
purging of out-of-retention segments without having to open the metadata.
**Testing**
Updated unit/integration tests.
I also tested and validated this behavior in a pre-production environment.
`feature` `bugfix`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
