Thanks for the suggestion. Both documentation and expansion time warnings are added in https://github.com/apache/beam/pull/34563 .
Yi On Mon, Mar 31, 2025 at 2:43 PM Kenneth Knowles <k...@apache.org> wrote: > One way that you might be able to reach some users is to issue a warning > in the code for SqlTransform.expand(), with the deprecation timeline. Maybe > this is under "etc" but I wanted to mention it because the other items were > all documentation. > > Kenn > > On Mon, Mar 31, 2025 at 1:34 PM Yi Hu via dev <dev@beam.apache.org> wrote: > >> Thanks for inputs! From discussion we have agreed on moving forward >> >> Next step: >> >> - Draft PR to note the deprecation status in Documentation, including >> CHANGES, Javadoc, etc >> >> - The earliest release to stop publishing ZetaSQL artifacts is pushed >> further, not earlier than 1 full quarter before the next Release (2.65.0 + >> 3 months would be 2.68.0), and only when it causes issues on maintenance. >> >> Thanks, >> >> Yi >> >> >> On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <k...@apache.org> wrote: >> >>> +1 to this deprecation. Thanks for putting together a clear summary. >>> >>> FWIW it also has significantly worse performance than Calcite SQL >>> dialect, since it calls out to a ZetaSQL subprocess for most calculations, >>> and that is less optimized than Beam's Fn API. >>> >>> Kenn >>> >>> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user < >>> u...@beam.apache.org> wrote: >>> >>>> I'm in favor of deprecating this and cleaning it up, but it depends on >>>> usage. I suspect it is low (or possibly non-existent, especially as there's >>>> little upside to moving away from the default). I cc'd user@ just in >>>> case anyone wants to chime in there. This may be a good thing to add to our >>>> release notes as well (perhaps we can get it in the one that's just about >>>> to go out). >>>> >>>> Unless there is strong, justified pushback, I'd get the deprecation >>>> status (e.g. on the javadocs, website) right away. For actual removal, I >>>> agree with the idea of waiting until it actually causes issues or we move >>>> to the next major beam release, though I might push back at 2.66 being a >>>> bit too quick even if the first condition is hit before then and might give >>>> people at least a quarter's notice. >>>> >>>> - Robert >>>> >>>> >>>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <dev@beam.apache.org> >>>> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL >>>>> component [1]. >>>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and >>>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons >>>>> >>>>> - Developments in Beam for ZetaSQL dialect effectively stalled since >>>>> early 2022 (See change history [3]) >>>>> >>>>> - Despite incomplete support status, there is no new bug / feature >>>>> request opened ever since we migrated to use GitHub Issue, suggesting >>>>> minimal adoption [4] >>>>> >>>>> - We still need to keep zetasql up-to-date if its dependency conflicts >>>>> with other google dependencies, as a result ZetaSQL component introduces >>>>> maintenance burden when upgrading GCP-BOM (e.g. [5]). >>>>> >>>>> - One of the main reason that using ZetaSQL dialect, per [2], was >>>>> because >>>>> >>>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful >>>>> in pipelines that write to or read from BigQuery tables. >>>>> >>>>> As of today, as GCP BigQuery now supports using GoogleSQL >>>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery >>>>> via BigQuery Connections API / Federated query [6, 7]. This largely >>>>> provides an alternative for using Beam's ZetaSQL interacting with >>>>> BigQuery. >>>>> >>>>> For these reasons, I propose initiating the process of deprecating >>>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made: >>>>> >>>>> Firstly, agree on when to document the deprecated status for ZetaSQL >>>>> component in javadoc, beam website, currently I recommend do it in the >>>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30, >>>>> 2025) >>>>> >>>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking >>>>> change, and I think we can leave the deprecated status as is until the >>>>> following situation emerges, whichever comes first, and no earlier than >>>>> Beam 2.66.0 (cut Jun 11, 2025) >>>>> >>>>> - Continued support for ZetaSQL component involving significant >>>>> burdens, like conflict with other Beam dependencies, supported Java >>>>> versions, etc, or >>>>> - When Beam moved to the next release major release (3) >>>>> >>>>> Thanks for your attention, and any input welcomed! >>>>> >>>>> Regards, >>>>> Yi >>>>> >>>>> [1] >>>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql >>>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/ >>>>> [3] >>>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java >>>>> [4] >>>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20 >>>>> [5] https://github.com/apache/beam/pull/32902 >>>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro >>>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro >>>>> >>>>> -- >>>>> >>>>> Yi Hu, (he/him/his) >>>>> >>>>> Software Engineer >>>>> >>>>> >>>>>
