David Mollitor created SPARK-60740:
--------------------------------------
Summary: Use ISO-8601 UTC timestamps in the default log4j2 layouts
Key: SPARK-60740
URL: https://issues.apache.org/jira/browse/SPARK-60740
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.3.0
Reporter: David Mollitor
Spark's default plain-text log layouts write timestamps as {{yy/MM/dd
HH:mm:ss}} in the JVM's local time zone. That format has a two-digit year and
no time zone, and on the console it has no milliseconds. This makes logs from
hosts in different time zones hard to correlate, and log aggregation tools
can't parse it without custom rules. It also doesn't match Spark's own JSON
structured-logging layout ({{{}SparkLayout.json{}}}), which already writes
ISO-8601 UTC timestamps.
Change the timestamp in the default layouts to
{{{}%d\{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}{UTC{}}}}:
||Layout||Used when||
|{{log4j2-defaults.properties}} (bundled)|No {{log4j2.properties}} is provided|
|{{conf/log4j2.properties.template}}|A user copies the template to
{{conf/log4j2.properties}}|
|{{DriverLogger}} default layout|Driver logs are written to
{{spark.driver.log.localDir}} / {{spark.driver.log.dfsDir}} and
{{spark.driver.log.layout}} is not set|
Example:
{noformat}
Before: 26/10/29 08:05:34 INFO SparkContext: Running Spark version ...
After: 2026-10-29T15:05:34.394Z INFO SparkContext: Running Spark version ...
{noformat}
Users who supply their own {{log4j2.properties}} are not affected. Because
existing log parsers may depend on the current format, this is a master-only
change for 5.0, documented in the core migration guide.
To keep the old format, provide a {{log4j2.properties}} that uses
{{{}%d{yy/MM/dd HH:mm:ss{}}}}, and set {{spark.driver.log.layout}} to
{{{}%d\{yy/MM/dd HH:mm:ss.SSS} %t %p %c\{1}: %m%n%ex{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
