[
https://issues.apache.org/jira/browse/SPARK-41794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-41794.
----------------------------------
Resolution: Fixed
Resolving as Fixed: this change was merged to master. Implementing commit:
8d8cc623085e [SPARK-41794][SQL] Add `try_remainder` function and re-enable
column tests. (Triaged as a stranded ticket — the fix landed but the JIRA was
left unresolved. Verified present on current master and not reverted.)
> Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_column
> -------------------------------------------------------------------
>
> Key: SPARK-41794
> URL: https://issues.apache.org/jira/browse/SPARK-41794
> Project: Spark
> Issue Type: Sub-task
> Components: Connect, Tests
> Affects Versions: 3.4.0
> Reporter: Hyukjin Kwon
> Assignee: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> ======================================================================
> ERROR [0.901s]: test_column_accessor
> (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/.../spark/python/pyspark/sql/tests/connect/test_connect_column.py",
> line 744, in test_column_accessor
> cdf.select(CF.col("z")[0], cdf.z[10], CF.col("z")[-10]).toPandas(),
> File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 949, in
> toPandas
> return self._session.client.to_pandas(query)
> File "/.../spark/python/pyspark/sql/connect/client.py", line 413, in
> to_pandas
> return self._execute_and_fetch(req)
> File "/.../spark/python/pyspark/sql/connect/client.py", line 573, in
> _execute_and_fetch
> self._handle_error(rpc_error)
> File "/.../spark/python/pyspark/sql/connect/client.py", line 623, in
> _handle_error
> raise SparkConnectException(status.message, info.reason) from None
> pyspark.sql.connect.client.SparkConnectException:
> (org.apache.spark.SparkArrayIndexOutOfBoundsException) [INVALID_ARRAY_INDEX]
> The index 10 is out of bounds. The array has 3 elements. Use the SQL function
> `get()` to tolerate accessing element at invalid index and return NULL
> instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this
> error.
> ======================================================================
> ERROR [0.245s]: test_column_arithmetic_ops
> (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/.../spark/python/pyspark/sql/tests/connect/test_connect_column.py",
> line 799, in test_column_arithmetic_ops
> cdf.select(cdf.a % cdf["b"], cdf["a"] % 2, 12 % cdf.c).toPandas(),
> File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 949, in
> toPandas
> return self._session.client.to_pandas(query)
> File "/.../spark/python/pyspark/sql/connect/client.py", line 413, in
> to_pandas
> return self._execute_and_fetch(req)
> File "/.../spark/python/pyspark/sql/connect/client.py", line 573, in
> _execute_and_fetch
> self._handle_error(rpc_error)
> File "/.../spark/python/pyspark/sql/connect/client.py", line 623, in
> _handle_error
> raise SparkConnectException(status.message, info.reason) from None
> pyspark.sql.connect.client.SparkConnectException:
> (org.apache.spark.SparkArithmeticException) [DIVIDE_BY_ZERO] Division by
> zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead.
> If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
