This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 969d2d4f627 [SPARK-40142][PYTHON][DOCS][FOLLOW-UP] Remove non-ANSI compliant example in element_at 969d2d4f627 is described below commit 969d2d4f62759f75125c0ee7e2324bda8927527c Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Thu Sep 22 11:44:05 2022 +0900 [SPARK-40142][PYTHON][DOCS][FOLLOW-UP] Remove non-ANSI compliant example in element_at ### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/37850 that removes non-ANSI compliant example in `element_at`. ### Why are the changes needed? ANSI build fails to run the example. https://github.com/apache/spark/actions/runs/3094607589/jobs/5008176959 ``` Caused by: org.apache.spark.SparkArrayIndexOutOfBoundsException: [INVALID_ARRAY_INDEX_IN_ELEMENT_AT] The index -4 is out of bounds. The array has 3 elements. Use `try_element_at` to tolerate accessing element at invalid index and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. at org.apache.spark.sql.errors.QueryExecutionErrors$.invalidElementAtIndexError(QueryExecutionErrors.scala:264) ... /usr/local/pypy/pypy3.7/lib-python/3/runpy.py:125: RuntimeWarning: 'pyspark.sql.functions' found in sys.modules after import of package 'pyspark.sql', but prior to execution of 'pyspark.sql.functions'; this may result in unpredictable behaviour warn(RuntimeWarning(msg)) /__w/spark/spark/python/pyspark/context.py:310: FutureWarning: Python 3.7 support is deprecated in Spark 3.4. warnings.warn("Python 3.7 support is deprecated in Spark 3.4.", FutureWarning) ********************************************************************** 1 of 6 in pyspark.sql.functions.element_at ``` ### Does this PR introduce _any_ user-facing change? No. The example added is not exposed to end users yet. ### How was this patch tested? Manually tested with enabling the ANSI configuration (`spark.sql.ansi.enabled`) Closes #37959 from HyukjinKwon/SPARK-40142-followup. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- python/pyspark/sql/functions.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py index 7885229fc5c..fe114e07c88 100644 --- a/python/pyspark/sql/functions.py +++ b/python/pyspark/sql/functions.py @@ -6595,11 +6595,6 @@ def element_at(col: "ColumnOrName", extraction: Any) -> Column: >>> df.select(element_at(df.data, -1)).collect() [Row(element_at(data, -1)='c')] - Returns `None` if there is no value corresponding to the given `extraction`. - - >>> df.select(element_at(df.data, -4)).collect() - [Row(element_at(data, -4)=None)] - >>> df = spark.createDataFrame([({"a": 1.0, "b": 2.0},)], ['data']) >>> df.select(element_at(df.data, lit("a"))).collect() [Row(element_at(data, a)=1.0)] --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org