codeant-ai-for-open-source[bot] commented on code in PR #41184: URL: https://github.com/apache/superset/pull/41184#discussion_r3565613580
########## superset/common/grouping_sets.py: ########## @@ -0,0 +1,117 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +SQL building blocks for the pivot-table non-additive totals optimization +(SIP.md, phase 3b). When a datasource engine reports +``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into +a single ``GROUPING SETS`` query: the database computes every level in one scan, +and each returned row is attributed to its level via ``GROUPING()`` markers. + +These are the engine-agnostic SQL primitives. Wiring them into the query context +(emitting one query and splitting the result back into per-level results) is the +remaining integration; see SIP.md. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from typing import Final + +import pandas as pd +from sqlalchemy import func, tuple_ +from sqlalchemy.sql.elements import ColumnElement + +# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS +# query. Chosen to be unlikely to collide with a real metric/column label. +GROUPING_MARKER_SUFFIX: Final = "__superset_grouping" + + +def grouping_marker_label(column_label: str) -> str: + """The output label of the GROUPING() marker for a groupby column.""" + return f"{column_label}{GROUPING_MARKER_SUFFIX}" + + +def grouping_sets_clause( + groups: Sequence[Sequence[ColumnElement]], +) -> ColumnElement: + """ + Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups. + + Each group is the set of columns grouped at one rollup level; the empty + group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]`` + produce ``GROUPING SETS ((a, b), (a), ())``. + + :param groups: one column list per rollup level + :return: a clause element suitable for ``select(...).group_by(...)`` + """ + return func.grouping_sets(*[tuple_(*group) for group in groups]) + + +def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement: + """ + Build a ``GROUPING(col) AS label`` marker column. + + In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is + part of the row's grouping level and ``1`` when it has been rolled up + (aggregated away). Selecting one marker per groupby column lets the caller + attribute each returned row to its rollup level when splitting the single + result back into per-level results. + + :param column: the groupby column to probe + :param label: the output label for the marker (see ``grouping_marker_label``) + :return: the labelled ``GROUPING(col)`` column + """ + return func.grouping(column).label(label) + + +def split_grouping_sets_result( + df: pd.DataFrame, + levels: Sequence[Sequence[str]], + groupby_columns: Sequence[str], +) -> list[pd.DataFrame]: + """ + Split a combined ``GROUPING SETS`` result into one DataFrame per rollup + level, the inverse of {@link grouping_sets_clause}. + + Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named + by ``grouping_marker_label``): ``0`` if the column is grouped at that row's + level, ``1`` if it was rolled up. A row belongs to a level iff its markers + are ``0`` exactly for that level's columns. Marker columns are dropped from + the returned frames so each looks like an ordinary per-level query result. + + :param df: the combined query result, including marker columns + :param levels: the grouped-column list for each rollup level (same order as + passed to ``grouping_sets_clause``) + :param groupby_columns: every groupby column that has a marker + :return: one DataFrame per level, in ``levels`` order + """ + markers: list[str] = [grouping_marker_label(col) for col in groupby_columns] + results: list[pd.DataFrame] = [] + for level in levels: + grouped: set[str] = set(level) + mask: pd.Series = pd.Series(True, index=df.index) + for col in groupby_columns: + expected = 0 if col in grouped else 1 Review Comment: **Suggestion:** Add an explicit type annotation for this local variable to satisfy the type-hint requirement for annotatable variables. [custom_rule] **Severity Level:** Minor ๐งน <details> <summary><b>Why it matters? โญ </b></summary> This new local variable has a stable, obvious type (`int`) and is introduced without an annotation. The stated rule requires type hints for annotatable variables, so this is a real violation. </details> <details> <summary><b>Rule source ๐ </b></summary> .cursor/rules/dev-standard.mdc (line 28) </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=29ba2337be984850a056b054700fcf35&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=29ba2337be984850a056b054700fcf35&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent ๐ค </b></summary> ```mdx This is a comment left during a code review. **Path:** superset/common/grouping_sets.py **Line:** 109:109 **Comment:** *Custom Rule: Add an explicit type annotation for this local variable to satisfy the type-hint requirement for annotatable variables. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=82120bfd60002996bfa32b74b394c6ac43ef73ed35e5d790e03f109a508dfa7e&reaction=like'>๐</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=82120bfd60002996bfa32b74b394c6ac43ef73ed35e5d790e03f109a508dfa7e&reaction=dislike'>๐</a> ########## superset/common/grouping_sets.py: ########## @@ -0,0 +1,117 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +SQL building blocks for the pivot-table non-additive totals optimization +(SIP.md, phase 3b). When a datasource engine reports +``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into +a single ``GROUPING SETS`` query: the database computes every level in one scan, +and each returned row is attributed to its level via ``GROUPING()`` markers. + +These are the engine-agnostic SQL primitives. Wiring them into the query context +(emitting one query and splitting the result back into per-level results) is the +remaining integration; see SIP.md. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from typing import Final + +import pandas as pd +from sqlalchemy import func, tuple_ +from sqlalchemy.sql.elements import ColumnElement + +# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS +# query. Chosen to be unlikely to collide with a real metric/column label. +GROUPING_MARKER_SUFFIX: Final = "__superset_grouping" + + +def grouping_marker_label(column_label: str) -> str: + """The output label of the GROUPING() marker for a groupby column.""" + return f"{column_label}{GROUPING_MARKER_SUFFIX}" + + +def grouping_sets_clause( + groups: Sequence[Sequence[ColumnElement]], +) -> ColumnElement: + """ + Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups. + + Each group is the set of columns grouped at one rollup level; the empty + group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]`` + produce ``GROUPING SETS ((a, b), (a), ())``. + + :param groups: one column list per rollup level + :return: a clause element suitable for ``select(...).group_by(...)`` + """ + return func.grouping_sets(*[tuple_(*group) for group in groups]) + + +def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement: + """ + Build a ``GROUPING(col) AS label`` marker column. + + In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is + part of the row's grouping level and ``1`` when it has been rolled up + (aggregated away). Selecting one marker per groupby column lets the caller + attribute each returned row to its rollup level when splitting the single + result back into per-level results. + + :param column: the groupby column to probe + :param label: the output label for the marker (see ``grouping_marker_label``) + :return: the labelled ``GROUPING(col)`` column + """ + return func.grouping(column).label(label) + + +def split_grouping_sets_result( + df: pd.DataFrame, + levels: Sequence[Sequence[str]], + groupby_columns: Sequence[str], +) -> list[pd.DataFrame]: + """ + Split a combined ``GROUPING SETS`` result into one DataFrame per rollup + level, the inverse of {@link grouping_sets_clause}. + + Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named + by ``grouping_marker_label``): ``0`` if the column is grouped at that row's + level, ``1`` if it was rolled up. A row belongs to a level iff its markers + are ``0`` exactly for that level's columns. Marker columns are dropped from + the returned frames so each looks like an ordinary per-level query result. + + :param df: the combined query result, including marker columns + :param levels: the grouped-column list for each rollup level (same order as + passed to ``grouping_sets_clause``) + :param groupby_columns: every groupby column that has a marker + :return: one DataFrame per level, in ``levels`` order + """ + markers: list[str] = [grouping_marker_label(col) for col in groupby_columns] + results: list[pd.DataFrame] = [] + for level in levels: + grouped: set[str] = set(level) + mask: pd.Series = pd.Series(True, index=df.index) + for col in groupby_columns: + expected = 0 if col in grouped else 1 + mask &= df[grouping_marker_label(col)] == expected + level_df = ( + df[mask] + .drop(columns=[m for m in markers if m in df.columns]) + .reset_index(drop=True) + ) Review Comment: **Suggestion:** Add a concrete type annotation to this local DataFrame variable assignment to keep local variable typing explicit. [custom_rule] **Severity Level:** Minor ๐งน <details> <summary><b>Why it matters? โญ </b></summary> The variable clearly holds a `pandas.DataFrame`, and it is introduced without a type annotation. That matches the custom requirement to annotate relevant variables that can be typed. </details> <details> <summary><b>Rule source ๐ </b></summary> .cursor/rules/dev-standard.mdc (line 28) </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=daf62a6903c04a0abf0902cffa544bd3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=daf62a6903c04a0abf0902cffa544bd3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent ๐ค </b></summary> ```mdx This is a comment left during a code review. **Path:** superset/common/grouping_sets.py **Line:** 111:115 **Comment:** *Custom Rule: Add a concrete type annotation to this local DataFrame variable assignment to keep local variable typing explicit. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=15766b4510bf1c5abcdcec23d3f9b71aa9f7e0f79084c907cd6b90e673c5a58e&reaction=like'>๐</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=15766b4510bf1c5abcdcec23d3f9b71aa9f7e0f79084c907cd6b90e673c5a58e&reaction=dislike'>๐</a> ########## superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts: ########## @@ -0,0 +1,446 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import { TimeGranularity } from '@superset-ui/core'; +import buildGroupbyCombinations, { + isAdditiveMetric, + allMetricsAdditive, + additiveReducerFor, + synthesizeAdditiveLevels, + splitGroupingSetsResult, + groupingMarkerLabel, +} from '../../src/plugin/utilities'; +import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types'; + +const baseFormData = { + groupbyRows: ['row1', 'row2'], + groupbyColumns: ['col1', 'col2'], + metrics: ['metric1', 'metric2'], + tableRenderer: 'Table With Subtotal', + colOrder: 'key_a_to_z', + rowOrder: 'key_a_to_z', + aggregateFunction: 'Sum', + metricsLayout: MetricsLayoutEnum.ROWS, + transposePivot: false, + rowSubtotalPosition: true, + colSubtotalPosition: true, + colTotals: true, + colSubTotals: true, + rowTotals: true, + rowSubTotals: true, + valueFormat: 'SMART_NUMBER', + datasource: '5__table', + viz_type: 'my_chart', + width: 800, + height: 600, + combineMetric: false, + verboseMap: {}, + columnFormats: {}, + currencyFormats: {}, + metricColorFormatters: [], + dateFormatters: {}, + setDataMask: () => {}, + legacy_order_by: 'count', + order_desc: true, + margin: 0, + time_grain_sqla: TimeGranularity.MONTH, + temporal_columns_lookup: { col1: true }, + currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' }, +} as unknown as PivotTableQueryFormData; + +test('should build all combinations for basic pivot table', () => { + const combinations = buildGroupbyCombinations(baseFormData); + + expect(combinations).toEqual([ + { rows: [], columns: [] }, + { rows: [], columns: ['col1'] }, + { rows: [], columns: ['col1', 'col2'] }, + { rows: ['row1'], columns: [] }, + { rows: ['row1'], columns: ['col1'] }, + { rows: ['row1'], columns: ['col1', 'col2'] }, + { rows: ['row1', 'row2'], columns: [] }, + { rows: ['row1', 'row2'], columns: ['col1'] }, + { rows: ['row1', 'row2'], columns: ['col1', 'col2'] }, + ]); + + expect(combinations).toHaveLength(9); +}); + +test('should handle transposed pivot correctly', () => { + const modifiedFormData = { + ...baseFormData, + transposePivot: true, + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([ + { rows: [], columns: [] }, + { rows: [], columns: ['row1'] }, + { rows: [], columns: ['row1', 'row2'] }, + { rows: ['col1'], columns: [] }, + { rows: ['col1'], columns: ['row1'] }, + { rows: ['col1'], columns: ['row1', 'row2'] }, + { rows: ['col1', 'col2'], columns: [] }, + { rows: ['col1', 'col2'], columns: ['row1'] }, + { rows: ['col1', 'col2'], columns: ['row1', 'row2'] }, + ]); +}); + +test('should filter combinations when combineMetric is true with ROWS layout', () => { + const modifiedFormData = { + ...baseFormData, + combineMetric: true, + metricsLayout: MetricsLayoutEnum.ROWS, + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([ + { rows: ['row1', 'row2'], columns: [] }, + { rows: ['row1', 'row2'], columns: ['col1'] }, + { rows: ['row1', 'row2'], columns: ['col1', 'col2'] }, + ]); + + expect(combinations).toHaveLength(3); +}); + +test('should filter combinations when combineMetric is true with COLUMNS layout', () => { + const modifiedFormData = { + ...baseFormData, + combineMetric: true, + metricsLayout: MetricsLayoutEnum.COLUMNS, + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([ + { rows: [], columns: ['col1', 'col2'] }, + { rows: ['row1'], columns: ['col1', 'col2'] }, + { rows: ['row1', 'row2'], columns: ['col1', 'col2'] }, + ]); + + expect(combinations).toHaveLength(3); +}); + +test('should handle single dimension in rows only', () => { + const modifiedFormData = { + ...baseFormData, + groupbyRows: ['row'], + groupbyColumns: [], + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([ + { rows: [], columns: [] }, + { rows: ['row'], columns: [] }, + ]); + + expect(combinations).toHaveLength(2); +}); + +test('should handle single dimension in columns only', () => { + const modifiedFormData = { + ...baseFormData, + groupbyRows: [], + groupbyColumns: ['col'], + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([ + { rows: [], columns: [] }, + { rows: [], columns: ['col'] }, + ]); + + expect(combinations).toHaveLength(2); +}); + +test('should handle empty groupby arrays', () => { + const modifiedFormData = { + ...baseFormData, + groupbyRows: [], + groupbyColumns: [], + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([{ rows: [], columns: [] }]); + + expect(combinations).toHaveLength(1); +}); + +test('should work with combineMetric and transposed pivot', () => { + const modifiedFormData = { + ...baseFormData, + transposePivot: true, + combineMetric: true, + metricsLayout: MetricsLayoutEnum.COLUMNS, + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([ + { rows: [], columns: ['row1', 'row2'] }, + { rows: ['col1'], columns: ['row1', 'row2'] }, + { rows: ['col1', 'col2'], columns: ['row1', 'row2'] }, + ]); +}); + +test('should handle combineMetric with empty arrays correctly', () => { + const modifiedFormData = { + ...baseFormData, + groupbyRows: [], + groupbyColumns: ['col'], + combineMetric: true, + metricsLayout: MetricsLayoutEnum.ROWS, + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toEqual([ + { rows: [], columns: [] }, + { rows: [], columns: ['col'] }, + ]); +}); + +test('should work with large number of dimensions', () => { + const modifiedFormData = { + ...baseFormData, + groupbyRows: ['r1', 'r2', 'r3', 'r4'], + groupbyColumns: ['c1', 'c2', 'c3'], + }; + + const combinations = buildGroupbyCombinations(modifiedFormData); + + expect(combinations).toHaveLength(20); + + expect(combinations).toContainEqual({ rows: [], columns: [] }); + expect(combinations).toContainEqual({ + rows: ['r1', 'r2', 'r3', 'r4'], + columns: ['c1', 'c2', 'c3'], + }); +}); + +test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => { + expect( + isAdditiveMetric({ + expressionType: 'SIMPLE', + aggregate: 'SUM', + column: { column_name: 'num' }, + label: 'sum_num', + } as any), + ).toBe(true); + expect( + isAdditiveMetric({ + expressionType: 'SIMPLE', + aggregate: 'COUNT', + column: { column_name: 'num' }, + label: 'count_num', + } as any), + ).toBe(true); +}); + +test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => { + expect( + isAdditiveMetric({ + expressionType: 'SIMPLE', + aggregate: 'AVG', + column: { column_name: 'num' }, + label: 'avg_num', + } as any), + ).toBe(false); Review Comment: **Suggestion:** Replace this `as any` cast with the appropriate typed metric shape for a non-additive aggregate case. [custom_rule] **Severity Level:** Minor ๐งน <details> <summary><b>Why it matters? โญ </b></summary> The code in the final file contains an `as any` cast here, so the no-any-types rule is actually violated. </details> <details> <summary><b>Rule source ๐ </b></summary> .cursor/rules/dev-standard.mdc (line 16) </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ff37acb4268e4e42a884813648358ba4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ff37acb4268e4e42a884813648358ba4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent ๐ค </b></summary> ```mdx This is a comment left during a code review. **Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts **Line:** 262:269 **Comment:** *Custom Rule: Replace this `as any` cast with the appropriate typed metric shape for a non-additive aggregate case. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fb81fed90482cb5b41e0cc07b221ef15e7676d153a58a2cffb0de66d2243367b&reaction=like'>๐</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fb81fed90482cb5b41e0cc07b221ef15e7676d153a58a2cffb0de66d2243367b&reaction=dislike'>๐</a> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
