[
https://issues.apache.org/jira/browse/BEAM-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuwei Fu updated BEAM-10379:
----------------------------
Description:
Performs a bitwise AND operation on expression and returns the result.
Supported Argument Types: INT64
Returned Data Types: INT64
Examples
{code:sql}
SELECT BIT_AND(c) as bit_and FROM UNNEST([0xF001, 0x00A1]) as c;
+---------+
| bit_and |
+---------+
| 1 |
+---------+
{code}
What is expected: should include both Calcite and ZetaSQL dialects.
How to test: unit tests
Reference:
[https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_and]
Problems:
After implementation, the current situation is:
1. When table is empty, the result is empty, correct.
2. When table contains only null values, the result is null, correct.
3. When table contains only valid numerical values, the result is as
expected, correct.
4. When table contains both valid numerical values and null values, the result
should be null (BitAnd(null, val) = null). But it seems all null values have
been directly ignored before doing the bit_and operation. Only numerical values
are taken into operation, so the result is incorrect.
It turns out that on direct runner, NULL will not be passed to CombineFn thus
all NULL inputs are ignored. And then if there is any non-null inputs, bit_and
will be applied on them only, which leads to a non-null result, which is not
correct.
Before figuring out the root cause and proposing a fix, the previous
implementation of BIT_AND is reverted.
was:
Performs a bitwise AND operation on expression and returns the result.
Supported Argument Types: INT64
Returned Data Types: INT64
Examples
{code:sql}
SELECT BIT_AND(c) as bit_and FROM UNNEST([0xF001, 0x00A1]) as c;
+---------+
| bit_and |
+---------+
| 1 |
+---------+
{code}
What is expected: should include both Calcite and ZetaSQL dialects.
How to test: unit tests
Reference:
[https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_and]
Problems:
After implementation, the current situation is:
1. When table is empty, the result is empty, correct.
2. When table contains only null values, the result is null, correct.
3. When table contains only valid numerical values, the result is as
expected, correct.
4. When table contains both valid numerical values and null values, the result
should be null (BitAnd(null, val) = null). But it seems all null values have
been directly ignored before doing the bit_and operation. Only numerical values
are taken into operation, so the result is incorrect.
It turns out that on direct runner, NULL will not be passed to CombineFn thus
all NULL inputs are ignored. And then if there is any non-null inputs, bit_and
will be applied on them only, which leads to a non-null result, which is not
correct.
> Some problems when implementing BIT_AND aggregation function in BeamSQL
> -----------------------------------------------------------------------
>
> Key: BEAM-10379
> URL: https://issues.apache.org/jira/browse/BEAM-10379
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, dsl-sql-zetasql
> Reporter: Yuwei Fu
> Priority: P2
> Labels: starter
>
> Performs a bitwise AND operation on expression and returns the result.
> Supported Argument Types: INT64
> Returned Data Types: INT64
> Examples
> {code:sql}
> SELECT BIT_AND(c) as bit_and FROM UNNEST([0xF001, 0x00A1]) as c;
> +---------+
> | bit_and |
> +---------+
> | 1 |
> +---------+
> {code}
>
> What is expected: should include both Calcite and ZetaSQL dialects.
> How to test: unit tests
> Reference:
> [https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_and]
>
> Problems:
> After implementation, the current situation is:
> 1. When table is empty, the result is empty, correct.
> 2. When table contains only null values, the result is null, correct.
> 3. When table contains only valid numerical values, the result is as
> expected, correct.
> 4. When table contains both valid numerical values and null values, the
> result should be null (BitAnd(null, val) = null). But it seems all null
> values have been directly ignored before doing the bit_and operation. Only
> numerical values are taken into operation, so the result is incorrect.
>
> It turns out that on direct runner, NULL will not be passed to CombineFn thus
> all NULL inputs are ignored. And then if there is any non-null inputs,
> bit_and will be applied on them only, which leads to a non-null result, which
> is not correct.
> Before figuring out the root cause and proposing a fix, the previous
> implementation of BIT_AND is reverted.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
