[
https://issues.apache.org/jira/browse/SPARK-28130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-28130:
---------------------------------
Description:
When {{xmlrunner}} is installed, seems {{xmlrunner}} does not respect
{{vervosity}} level in unittest.
So the output looks as below
{code}
02:54:42 Running tests...
02:54:42
----------------------------------------------------------------------
02:54:42 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
02:54:42
----------------------------------------------------------------------
02:54:42 Ran 48 tests in 0.002s
02:54:42
02:54:42 OK (skipped=48)
{code}
and it is not caught by our message detection mechanism added by SPARK-24044
If we manually set the {{vervocity}} level to {{xmlrunner}}, it prints messages
as below:
{code}
02:54:42 Running tests...
02:54:42
----------------------------------------------------------------------
02:54:42 test_datasource_with_udf
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_mixed_udf
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_mixed_udf_and_sql
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_nondeterministic_vectorized_udf
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_nondeterministic_vectorized_udf_in_aggregate
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_pandas_udf_nested_arrays
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_pandas_udf_tokenize
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_register_nondeterministic_vectorized_udf_basic
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_register_vectorized_udf_basic
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_timestamp_dst
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_type_annotation
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
{code}
Note that last {{SKIP}} is different. Looks we should fix the regular
expression to catch {{SKIP}} case as well.
was:
When {{xmlrunner}} is installed, seems {{xmlrunner}} does not respect
{{vervosity}} level in unittest.
So the output looks as below
{code}
02:54:42 Running tests...
02:54:42
----------------------------------------------------------------------
02:54:42 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
02:54:42
----------------------------------------------------------------------
02:54:42 Ran 48 tests in 0.002s
02:54:42
02:54:42 OK (skipped=48)
{code}
and it is not caught by our message detection mechanism added by SPARK-24044
If we manually set the {{vervocity}} level to {{xmlrunner}}, it prints messages
as below:
{code}
{code}
02:54:42 Running tests...
02:54:42
----------------------------------------------------------------------
02:54:42 test_datasource_with_udf
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_mixed_udf
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_mixed_udf_and_sql
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_nondeterministic_vectorized_udf
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_nondeterministic_vectorized_udf_in_aggregate
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_pandas_udf_nested_arrays
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_pandas_udf_tokenize
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_register_nondeterministic_vectorized_udf_basic
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_register_vectorized_udf_basic
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_timestamp_dst
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
02:54:42 test_type_annotation
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
(0.000s)
{code}
Note that last {{SKIP}} is different. Looks we should fix the regular
expression to catch {{SKIP}} case as well.
> Pretty messages not being printed for skipped PySpark tests when xmlrunner is
> available
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-28130
> URL: https://issues.apache.org/jira/browse/SPARK-28130
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra, PySpark
> Affects Versions: 3.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> When {{xmlrunner}} is installed, seems {{xmlrunner}} does not respect
> {{vervosity}} level in unittest.
> So the output looks as below
> {code}
> 02:54:42 Running tests...
> 02:54:42
> ----------------------------------------------------------------------
> 02:54:42 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
> 02:54:42
> ----------------------------------------------------------------------
> 02:54:42 Ran 48 tests in 0.002s
> 02:54:42
> 02:54:42 OK (skipped=48)
> {code}
> and it is not caught by our message detection mechanism added by SPARK-24044
> If we manually set the {{vervocity}} level to {{xmlrunner}}, it prints
> messages as below:
> {code}
> 02:54:42 Running tests...
> 02:54:42
> ----------------------------------------------------------------------
> 02:54:42 test_datasource_with_udf
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_mixed_udf
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_mixed_udf_and_sql
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_nondeterministic_vectorized_udf
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_nondeterministic_vectorized_udf_in_aggregate
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_pandas_udf_nested_arrays
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_pandas_udf_tokenize
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_register_nondeterministic_vectorized_udf_basic
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_register_vectorized_udf_basic
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_timestamp_dst
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> 02:54:42 test_type_annotation
> (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ... SKIP
> (0.000s)
> {code}
> Note that last {{SKIP}} is different. Looks we should fix the regular
> expression to catch {{SKIP}} case as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
