andygrove opened a new pull request, #7287: URL: https://github.com/apache/datafusion-comet/pull/7287
## Which issue does this PR close? Part of #7204. Documents #7203, #5022 and #6936 instead of fixing them in 1.2.0. ## Rationale for this change #7203, #5022 and #6936 are differences from Spark that 1.1.0 did not have, found in the audit tracked in #7204. None has a fix, and none will be in 1.2.0, so the upgrade guide tells users about them and gives the setting that avoids each one. ## What changes are included in this PR? A new section, "Known Differences From Spark That Are New in 1.2.0", in `docs/source/user-guide/latest/migration-guide.md` under "Upgrading to Comet 1.2.0". It covers: - #7203: `collect_set` over `FLOAT` or `DOUBLE` next to `count(DISTINCT ...)` on Spark before 4.2 (signed zeros and `NaN`). - #5022: a `RANGE` window frame over a `DATE` column with an offset of about 92 million days or more. - #6936: native `STRING` and `BINARY` `min` and `max` under-reporting memory in a hash aggregate. The guide says it leaves out correctness fixes, so the section's opening paragraph explains why these three are listed. ## How are these changes tested? Prettier passes on the file. The text comes from the issues, the code on `branch-1.2` and the audit in #7204. I did not run a query for any of them on this branch, so these points are unchecked: - `spark.comet.exec.strictFloatingPoint=true` should make the `count(DISTINCT ...)` shape fall back for #7203. It is the existing gate in `CometCollectSet.getSupportLevel`, but I did not run that query. - The roughly 92 million day threshold for #5022 comes from the estimate in #7204. The code falls back above 2,144,550,751 days. - The `spark.sql.execution.useObjectHashAggregateExec=false` setting for #6936 is the workaround given in the issue. `main` has an "Upgrading to Comet 1.2.0" section too. It needs the same text, so a counterpart PR against `main` is still to do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
