yashmayya opened a new pull request, #14264: URL: https://github.com/apache/pinot/pull/14264
- The SQL standard defines a `RESPECT NULLS` or `IGNORE NULLS` option for the window functions `LEAD`, `LAG`, `FIRST_VALUE`, `LAST_VALUE`, and `NTH_VALUE` (although Pinot currently doesn't support this function). The default behavior is `RESPECT NULLS`. - This patch adds support for these options on the `FIRST_VALUE` and `LAST_VALUE` window functions (`LEAD` / `LAG` can be added in a subsequent patch). As the name suggests, the `IGNORE NULLS` option makes it so that the `FIRST_VALUE` and `LAST_VALUE` window functions compute the first and last non-null values respectively for each window frame. - If `IGNORE NULLS` is specified like `LAST_VALUE(col1) IGNORE NULLS OVER (ORDER BY ts)`, it can effectively be used to gapfill data (see this article for example - https://learn.microsoft.com/en-us/azure/azure-sql-edge/imputing-missing-values). - Calcite has validation to ensure that the `IGNORE NULLS` / `RESPECT NULLS` operators are only used with window functions that they are applicable to as per standard SQL. This patch also updates the operators being registered in Pinot's operator table for `LEAD` / `LAG` since we don't currently support the null related options for those functions (this way, we fail during query planning rather than at runtime). - There are also some minor changes to the query plan serde here to hold the `IGNORE NULLS` option for a window function call. - The changes in this patch are built on top of https://github.com/apache/pinot/pull/14249. This PR will be rebased once that one is merged. Until then, the diff between the two can be reviewed more easily here - https://github.com/yashmayya/pinot/pull/1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
