This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b6656afa4e5 [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import
`WriteOperation` only when `pandas` is available
b6656afa4e5 is described below
commit b6656afa4e54e489c4642522ec74d850f53d3a83
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Dec 6 03:03:17 2022 -0800
[SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import `WriteOperation` only when
`pandas` is available
### What changes were proposed in this pull request?
This is the last piece to recover `pyspark-connect` tests on a system where
pandas is unavailable.
### Why are the changes needed?
**BEFORE**
```
$ python/run-tests.py --modules pyspark-connect
...
ModuleNotFoundError: No module named 'pandas'
```
**AFTER**
```
$ python/run-tests.py --modules pyspark-connect
Running PySpark tests. Output is in
/Users/dongjoon/APACHE/spark-merge/python/unit-tests.log
Will test against the following Python executables: ['python3.9']
Will test the following Python modules: ['pyspark-connect']
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.14
Starting test(python3.9):
pyspark.sql.tests.connect.test_connect_column_expressions (temp output:
/Users/dongjoon/APACHE/spark-merge/python/target/fc8f073e-edbe-470f-a3cc-f35b3f4b5262/python3.9__pyspark.sql.tests.connect.test_connect_column_expressions___lzz5fky.log)
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_basic
(temp output:
/Users/dongjoon/APACHE/spark-merge/python/target/c14dfac4-14df-45b4-b54e-6dbc3c58f128/python3.9__pyspark.sql.tests.connect.test_connect_basic__x8rz2dz1.log)
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_column
(temp output:
/Users/dongjoon/APACHE/spark-merge/python/target/5810e639-75c1-4e6c-91b3-2a1894dab319/python3.9__pyspark.sql.tests.connect.test_connect_column__3e_qmue7.log)
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_function
(temp output:
/Users/dongjoon/APACHE/spark-merge/python/target/8bebb3be-c146-4030-96b3-82ff7a873d49/python3.9__pyspark.sql.tests.connect.test_connect_function__hwe9aca8.log)
Finished test(python3.9):
pyspark.sql.tests.connect.test_connect_column_expressions (1s) ... 9 tests were
skipped
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_plan_only
(temp output:
/Users/dongjoon/APACHE/spark-merge/python/target/97cbccca-1e9a-44d9-958f-e2368266b437/python3.9__pyspark.sql.tests.connect.test_connect_plan_only__dffs5ux1.log)
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_column
(1s) ... 2 tests were skipped
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_select_ops
(temp output:
/Users/dongjoon/APACHE/spark-merge/python/target/297a054f-c577-4aee-b1ff-48d48beb423c/python3.9__pyspark.sql.tests.connect.test_connect_select_ops__4e__w6gh.log)
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_function
(1s) ... 5 tests were skipped
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_basic (1s)
... 48 tests were skipped
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_select_ops
(0s) ... 2 tests were skipped
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_plan_only
(1s) ... 28 tests were skipped
Tests passed in 2 seconds
Skipped tests in pyspark.sql.tests.connect.test_connect_basic with
python3.9:
test_channel_properties
(pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip
(0.002s)
test_invalid_connection_strings
(pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip
(0.000s)
test_metadata
(pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip
(0.000s)
test_sensible_defaults
(pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip
(0.000s)
test_valid_channel_creation
(pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip
(0.000s)
test_agg_with_avg
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_agg_with_two_agg_exprs
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_collect
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_count
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_create_global_temp_view
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_create_session_local_temp_view
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_crossjoin
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_deduplicate
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_drop
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_drop_na
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
test_empty_dataset
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_explain_string
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_fill_na
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
test_first
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_head
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_input_files
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_is_empty
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_is_local
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_is_streaming
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_join_condition_column_list_columns
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
test_limit_offset
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_print_schema
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_range
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
test_replace
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
test_repr
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_schema
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
test_select_expr
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_session
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_show
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_simple_datasource_read
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_simple_explain_string
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_simple_read
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_simple_udf
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_sort
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
test_sql
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_subquery_alias
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_tail
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_take
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_toDF
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_to_pandas
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_with_columns
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_with_columns_renamed
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.000s)
test_write_operations
(pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip
(0.001s)
Skipped tests in pyspark.sql.tests.connect.test_connect_column with
python3.9:
test_column_operator
(pyspark.sql.tests.connect.test_connect_column.SparkConnectTests) ... skip
(0.002s)
test_columns
(pyspark.sql.tests.connect.test_connect_column.SparkConnectTests) ... skip
(0.001s)
Skipped tests in pyspark.sql.tests.connect.test_connect_column_expressions
with python3.9:
test_binary_literal
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.002s)
test_column_alias
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.000s)
test_column_literals
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.000s)
test_float_nan_inf
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.000s)
test_map_literal
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.001s)
test_null_literal
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.000s)
test_numeric_literal_types
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.000s)
test_simple_column_expressions
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.000s)
test_uuid_literal
(pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite)
... skip (0.000s)
Skipped tests in pyspark.sql.tests.connect.test_connect_function with
python3.9:
test_aggregation_functions
(pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ...
skip (0.006s)
test_math_functions
(pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ...
skip (0.025s)
test_normal_functions
(pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ...
skip (0.002s)
test_sort_with_nulls_order
(pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ...
skip (0.001s)
test_sorting_functions_with_column
(pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ...
skip (0.000s)
Skipped tests in pyspark.sql.tests.connect.test_connect_plan_only with
python3.9:
test_all_the_plans
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.013s)
test_coalesce_and_repartition
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.002s)
test_crossjoin
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_crosstab
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_datasource_read
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_deduplicate
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_drop
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_drop_na
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_except
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_fill_na
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_filter
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_intersect
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_join_condition
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_join_using_columns
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_limit
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_offset
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_range
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_relation_alias
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_replace
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_sample
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_simple_project
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_simple_udf
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.000s)
test_sort
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_sql_project
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.002s)
test_summary
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_union
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.002s)
test_unsupported_functions
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
test_write_operation
(pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly)
... skip (0.001s)
Skipped tests in pyspark.sql.tests.connect.test_connect_select_ops with
python3.9:
test_join_with_join_type
(pyspark.sql.tests.connect.test_connect_select_ops.SparkConnectToProtoSuite)
... skip (0.085s)
test_select_with_columns_and_strings
(pyspark.sql.tests.connect.test_connect_select_ops.SparkConnectToProtoSuite)
... skip (0.000s)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the CIs.
Closes #38934 from dongjoon-hyun/SPARK-41317.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/sql/tests/connect/test_connect_plan_only.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/tests/connect/test_connect_plan_only.py
b/python/pyspark/sql/tests/connect/test_connect_plan_only.py
index 590c90aef97..0d52e66afd0 100644
--- a/python/pyspark/sql/tests/connect/test_connect_plan_only.py
+++ b/python/pyspark/sql/tests/connect/test_connect_plan_only.py
@@ -17,13 +17,13 @@
from typing import cast
import unittest
-from pyspark.sql.connect.plan import WriteOperation
from pyspark.testing.connectutils import PlanOnlyTestFixture
from pyspark.testing.sqlutils import have_pandas, pandas_requirement_message
if have_pandas:
import pyspark.sql.connect.proto as proto
from pyspark.sql.connect.column import Column
+ from pyspark.sql.connect.plan import WriteOperation
from pyspark.sql.connect.readwriter import DataFrameReader
from pyspark.sql.connect.function_builder import UserDefinedFunction, udf
from pyspark.sql.types import StringType
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
