This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new e07e291cc7a [SPARK-44929][TESTS] Standardize log output for console
appender in tests
e07e291cc7a is described below
commit e07e291cc7a325e7b698d1a974ff68423e1781e5
Author: Kent Yao <y...@apache.org>
AuthorDate: Thu Aug 24 13:51:26 2023 +0800
[SPARK-44929][TESTS] Standardize log output for console appender in tests
### What changes were proposed in this pull request?
This PR set a character length limit for the error message and a stack
depth limit for error stack traces to the console appender in tests.
The original patterns are
- %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
- %t: %m%n%ex
And they're adjusted to the new consistent pattern
- `%d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n`
### Why are the changes needed?
In testing, intentional and unintentional failures are created to generate
extensive log volumes. For instance, a single FileNotFound error may be logged
multiple times in the writer, task runner, task set manager, and other areas,
resulting in thousands of lines per failure.
For example, tests in ParquetRebaseDatetimeSuite will be run with V1 and V2
Datasource, two or more specific values, and multiple configuration pairs. I
have seen the SparkUpgradeException all over the CI logs
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
```
build/sbt "sql/testOnly *ParquetRebaseDatetimeV1Suite"
```
```
15:59:55.446 ERROR
org.apache.spark.sql.execution.datasources.FileFormatWriter: Job
job_202308230059551630377040190578321_1301 aborted.
15:59:55.446 ERROR org.apache.spark.executor.Executor: Exception in task
0.0 in stage 1301.0 (TID 1595)
org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while
writing rows to
file:/Users/hzyaoqin/spark/target/tmp/spark-67cce58e-dfb2-4811-a9c0-50ec4c90d1f1.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:765)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420)
at
org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
15:59:55.446 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0
in stage 1301.0 (TID 1595) (10.221.97.38 executor driver):
org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing
rows to
file:/Users/hzyaoqin/spark/target/tmp/spark-67cce58e-dfb2-4811-a9c0-50ec4c90d1f1.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:765)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420)
at org.apache.spark.sql.execution.datasources....
15:59:55.446 ERROR org.apache.spark.scheduler.TaskSetManager: Task 0 in
stage 1301.0 failed 1 times; aborting job
15:59:55.447 ERROR
org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job
0ead031e-c9dd-446b-b20b-c76ec54978b1.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 1301.0 failed 1 times, most recent failure: Lost task 0.0 in stage
1301.0 (TID 1595) (10.221.97.38 executor driver):
org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing
rows to
file:/Users/hzyaoqin/spark/target/tmp/spark-67cce58e-dfb2-4811-a9c0-50ec4c90d1f1.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:765)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420)
at
org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
15:59:55.579 ERROR org.apache.spark.executor.Executor: Exception in task
0.0 in stage 1303.0 (TID 1597)
```
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #42627 from yaooqinn/SPARK-44929.
Authored-by: Kent Yao <y...@apache.org>
Signed-off-by: Kent Yao <y...@apache.org>
(cherry picked from commit 830500150f7e3972d1fa5b47d0ab564bfa7e4b12)
Signed-off-by: Kent Yao <y...@apache.org>
---
connector/connect/client/jvm/src/test/resources/log4j2.properties | 2 +-
connector/connect/server/src/test/resources/log4j2.properties | 2 +-
connector/protobuf/src/test/resources/log4j2.properties | 2 +-
core/src/test/resources/log4j2.properties | 2 +-
hadoop-cloud/src/test/resources/log4j2.properties | 2 +-
repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala | 2 +-
.../src/test/resources/log-config-test-log4j.properties | 2 +-
.../org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala | 2 +-
.../test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 2 +-
sql/core/src/test/resources/log4j2.properties | 2 +-
sql/hive-thriftserver/src/test/resources/log4j2.properties | 2 +-
.../apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala | 2 +-
.../scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala | 2 +-
sql/hive/src/test/resources/log4j2.properties | 2 +-
14 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/connector/connect/client/jvm/src/test/resources/log4j2.properties
b/connector/connect/client/jvm/src/test/resources/log4j2.properties
index ab02104c696..550fd261b6f 100644
--- a/connector/connect/client/jvm/src/test/resources/log4j2.properties
+++ b/connector/connect/client/jvm/src/test/resources/log4j2.properties
@@ -32,7 +32,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %t: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/connector/connect/server/src/test/resources/log4j2.properties
b/connector/connect/server/src/test/resources/log4j2.properties
index ac4c1de0e91..f782d7f3aaa 100644
--- a/connector/connect/server/src/test/resources/log4j2.properties
+++ b/connector/connect/server/src/test/resources/log4j2.properties
@@ -32,7 +32,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %t: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/connector/protobuf/src/test/resources/log4j2.properties
b/connector/protobuf/src/test/resources/log4j2.properties
index ab02104c696..550fd261b6f 100644
--- a/connector/protobuf/src/test/resources/log4j2.properties
+++ b/connector/protobuf/src/test/resources/log4j2.properties
@@ -32,7 +32,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %t: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/core/src/test/resources/log4j2.properties
b/core/src/test/resources/log4j2.properties
index ab02104c696..550fd261b6f 100644
--- a/core/src/test/resources/log4j2.properties
+++ b/core/src/test/resources/log4j2.properties
@@ -32,7 +32,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %t: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/hadoop-cloud/src/test/resources/log4j2.properties
b/hadoop-cloud/src/test/resources/log4j2.properties
index 01a9cafafa8..215cc8e4888 100644
--- a/hadoop-cloud/src/test/resources/log4j2.properties
+++ b/hadoop-cloud/src/test/resources/log4j2.properties
@@ -33,7 +33,7 @@ appender.console.type = Console
appender.console.name = STDERR
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %t: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark_project.jetty
diff --git a/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
b/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
index 1f44826c100..bb2a85cfa0d 100644
--- a/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
+++ b/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
@@ -284,7 +284,7 @@ class ReplSuite extends SparkFunSuite {
|appender.console.target = SYSTEM_ERR
|appender.console.follow = true
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n%ex
+ |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
|
|# Set the log level for this class to ERROR same as the default
setting.
|logger.repl.name = org.apache.spark.repl.Main
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties
b/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties
index 4aeea886251..c30c42e1ebd 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties
+++
b/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties
@@ -23,4 +23,4 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
index aea4486e0e2..743de91b08f 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
@@ -47,7 +47,7 @@ private[spark] trait DecommissionSuite { k8sSuite:
KubernetesSuite =>
|appender.console.name = console
|appender.console.target = SYSTEM_OUT
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n%ex
+ |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
|
|logger.spark.name = org.apache.spark
|logger.spark.level = debug
diff --git
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index 472f29d1117..d166c50a1dd 100644
---
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -51,7 +51,7 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite
with Matchers {
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
+ |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
|logger.jetty.name = org.sparkproject.jetty
|logger.jetty.level = warn
|logger.eclipse.name = org.eclipse.jetty
diff --git a/sql/core/src/test/resources/log4j2.properties
b/sql/core/src/test/resources/log4j2.properties
index 49df23c4fc7..7ab47c16d4f 100644
--- a/sql/core/src/test/resources/log4j2.properties
+++ b/sql/core/src/test/resources/log4j2.properties
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = warn
diff --git a/sql/hive-thriftserver/src/test/resources/log4j2.properties
b/sql/hive-thriftserver/src/test/resources/log4j2.properties
index cc483f4671e..ebb3a6ccb2f 100644
--- a/sql/hive-thriftserver/src/test/resources/log4j2.properties
+++ b/sql/hive-thriftserver/src/test/resources/log4j2.properties
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
appender.console.filter.1.type = Filters
diff --git
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
index 8fd432148bb..b421a94a06c 100644
---
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
+++
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
@@ -1229,7 +1229,7 @@ abstract class HiveThriftServer2TestBase extends
SparkFunSuite with BeforeAndAft
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n%ex
+ |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
""".stripMargin,
new File(s"$tempLog4jConf/log4j2.properties"),
StandardCharsets.UTF_8)
diff --git
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala
index 02ee8dd273d..53a69d964b1 100644
---
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala
+++
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala
@@ -80,7 +80,7 @@ class UISeleniumSuite
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n%ex
+ |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
""".stripMargin,
new File(s"$tempLog4jConf/log4j2.properties"),
StandardCharsets.UTF_8)
diff --git a/sql/hive/src/test/resources/log4j2.properties
b/sql/hive/src/test/resources/log4j2.properties
index 6e2c01c4b14..c91f417bc6b 100644
--- a/sql/hive/src/test/resources/log4j2.properties
+++ b/sql/hive/src/test/resources/log4j2.properties
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c:
%maxLen{%m}{512}%n%ex{8}%n
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = warn
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
