[jira] [Commented] (SPARK-47599) MLLib: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844869#comment-17844869 ] Gengliang Wang commented on SPARK-47599: I checked with [~itholic] and he is not available for this one. [~panbingkun] are you interested in this one? > MLLib: Migrate logWarn with variables to structured logging framework > - > > Key: SPARK-47599 > URL: https://issues.apache.org/jira/browse/SPARK-47599 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48182) SQL (java side): Migrate `error/warn/info` with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-48182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48182. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46450 [https://github.com/apache/spark/pull/46450] > SQL (java side): Migrate `error/warn/info` with variables to structured > logging framework > - > > Key: SPARK-48182 > URL: https://issues.apache.org/jira/browse/SPARK-48182 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48182) SQL (java side): Migrate `error/warn/info` with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-48182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-48182: -- Assignee: BingKun Pan > SQL (java side): Migrate `error/warn/info` with variables to structured > logging framework > - > > Key: SPARK-48182 > URL: https://issues.apache.org/jira/browse/SPARK-48182 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Critical > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48126) Make spark.log.structuredLogging.enabled effective
[ https://issues.apache.org/jira/browse/SPARK-48126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48126. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46452 [https://github.com/apache/spark/pull/46452] > Make spark.log.structuredLogging.enabled effective > -- > > Key: SPARK-48126 > URL: https://issues.apache.org/jira/browse/SPARK-48126 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Currently, the spark conf spark.log.structuredLogging.enabled is not taking > effects. We need to fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48126) Make spark.log.structuredLogging.enabled effective
[ https://issues.apache.org/jira/browse/SPARK-48126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-48126: --- Summary: Make spark.log.structuredLogging.enabled effective (was: Make spark.log.structuredLogging.enabled effecitve) > Make spark.log.structuredLogging.enabled effective > -- > > Key: SPARK-48126 > URL: https://issues.apache.org/jira/browse/SPARK-48126 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > > Currently, the spark conf spark.log.structuredLogging.enabled is not taking > effects. We need to fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48134) Spark core (java side): Migrate `error/warn/info` with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-48134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48134. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46390 [https://github.com/apache/spark/pull/46390] > Spark core (java side): Migrate `error/warn/info` with variables to > structured logging framework > > > Key: SPARK-48134 > URL: https://issues.apache.org/jira/browse/SPARK-48134 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48134) Spark core (java side): Migrate `error/warn/info` with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-48134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-48134: -- Assignee: BingKun Pan > Spark core (java side): Migrate `error/warn/info` with variables to > structured logging framework > > > Key: SPARK-48134 > URL: https://issues.apache.org/jira/browse/SPARK-48134 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Critical > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48145) Remove logDebug and logTrace with MDC in java structured logging framework
Gengliang Wang created SPARK-48145: -- Summary: Remove logDebug and logTrace with MDC in java structured logging framework Key: SPARK-48145 URL: https://issues.apache.org/jira/browse/SPARK-48145 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48131) Unify MDC key `mdc.taskName` and `task_name`
Gengliang Wang created SPARK-48131: -- Summary: Unify MDC key `mdc.taskName` and `task_name` Key: SPARK-48131 URL: https://issues.apache.org/jira/browse/SPARK-48131 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang Rename the MDC key `mdc.taskName` as `task_name`, so that it is consistent with all the MDC keys used in the structured logging framework. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48129) Provide a constant table schema in PySpark for querying structured logs
Gengliang Wang created SPARK-48129: -- Summary: Provide a constant table schema in PySpark for querying structured logs Key: SPARK-48129 URL: https://issues.apache.org/jira/browse/SPARK-48129 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48123) Provide a constant table schema for querying structured logs
[ https://issues.apache.org/jira/browse/SPARK-48123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48123. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46375 [https://github.com/apache/spark/pull/46375] > Provide a constant table schema for querying structured logs > > > Key: SPARK-48123 > URL: https://issues.apache.org/jira/browse/SPARK-48123 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Providing a table schema LOG_SCHEMA, so that users can load structured logs > with the following: > ``` > spark.read.schema(LOG_SCHEMA).json(logPath) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47578) Spark core: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47578. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46309 [https://github.com/apache/spark/pull/46309] > Spark core: Migrate logWarn with variables to structured logging framework > -- > > Key: SPARK-47578 > URL: https://issues.apache.org/jira/browse/SPARK-47578 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Daniel >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48124) Disable structured logging for Interpreter by default
[
https://issues.apache.org/jira/browse/SPARK-48124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-48124:
---
Description:
Since there are plain text output from
Interpreters(spark-shell/spark-sql/pyspark), it makes more sense to disable
structured logging for Interpreters by default.
spark-shell output when with structured logging enabled:
```
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9)
Type in expressions to have them evaluated.
Type :help for more information.
{"ts":"2024-05-04T01:11:03.797Z","level":"WARN","msg":"Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable","logger":"NativeCodeLoader"}
{"ts":"2024-05-04T01:11:04.104Z","level":"WARN","msg":"Service 'SparkUI' could
not bind on port 4040. Attempting port 4041.","logger":"Utils"}
Spark context Web UI available at http://10.10.114.155:4041
Spark context available as 'sc' (master = local[*], app id =
local-1714785064155).
Spark session available as 'spark'.
```
spark-shell output when without structured logging enabled:
```
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9)
Type in expressions to have them evaluated.
Type :help for more information.
24/05/03 18:11:35 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
24/05/03 18:11:35 WARN Utils: Service 'SparkUI' could not bind on port 4040.
Attempting port 4041.
Spark context Web UI available at http://10.10.114.155:4041
Spark context available as 'sc' (master = local[*], app id =
local-1714785095892).
Spark session available as 'spark'.
```
was:
Since there are plain text output from
Interpreters(spark-shell/spark-sql/pyspark), it makes more sense to disable
structured logging for Interpreters by default.
> Disable structured logging for Interpreter by default
> -
>
> Key: SPARK-48124
> URL: https://issues.apache.org/jira/browse/SPARK-48124
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
>
> Since there are plain text output from
> Interpreters(spark-shell/spark-sql/pyspark), it makes more sense to disable
> structured logging for Interpreters by default.
>
> spark-shell output when with structured logging enabled:
> ```
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
> /_/
>
> Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9)
> Type in expressions to have them evaluated.
> Type :help for more information.
> {"ts":"2024-05-04T01:11:03.797Z","level":"WARN","msg":"Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable","logger":"NativeCodeLoader"}
> {"ts":"2024-05-04T01:11:04.104Z","level":"WARN","msg":"Service 'SparkUI'
> could not bind on port 4040. Attempting port 4041.","logger":"Utils"}
> Spark context Web UI available at http://10.10.114.155:4041
> Spark context available as 'sc' (master = local[*], app id =
> local-1714785064155).
> Spark session available as 'spark'.
> ```
>
> spark-shell output when without structured logging enabled:
> ```
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
> /_/
>
> Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9)
> Type in expressions to have them evaluated.
> Type :help for more information.
> 24/05/03 18:11:35 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 24/05/03 18:11:35 WARN Utils:
[jira] [Updated] (SPARK-48124) Disable structured logging for Interpreter by default
[ https://issues.apache.org/jira/browse/SPARK-48124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-48124: --- Description: Since there are plain text output from Interpreters(spark-shell/spark-sql/pyspark), it makes more sense to disable structured logging for Interpreters by default. > Disable structured logging for Interpreter by default > - > > Key: SPARK-48124 > URL: https://issues.apache.org/jira/browse/SPARK-48124 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > > Since there are plain text output from > Interpreters(spark-shell/spark-sql/pyspark), it makes more sense to disable > structured logging for Interpreters by default. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48126) Make spark.log.structuredLogging.enabled effecitve
Gengliang Wang created SPARK-48126: -- Summary: Make spark.log.structuredLogging.enabled effecitve Key: SPARK-48126 URL: https://issues.apache.org/jira/browse/SPARK-48126 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang Currently, the spark conf spark.log.structuredLogging.enabled is not taking effects. We need to fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48124) Disable structured logging for Interpreter by default
Gengliang Wang created SPARK-48124: -- Summary: Disable structured logging for Interpreter by default Key: SPARK-48124 URL: https://issues.apache.org/jira/browse/SPARK-48124 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48059) Implement the structured log framework on the java side
[ https://issues.apache.org/jira/browse/SPARK-48059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-48059: -- Assignee: BingKun Pan > Implement the structured log framework on the java side > --- > > Key: SPARK-48059 > URL: https://issues.apache.org/jira/browse/SPARK-48059 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Critical > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48059) Implement the structured log framework on the java side
[ https://issues.apache.org/jira/browse/SPARK-48059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48059. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46301 [https://github.com/apache/spark/pull/46301] > Implement the structured log framework on the java side > --- > > Key: SPARK-48059 > URL: https://issues.apache.org/jira/browse/SPARK-48059 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48123) Provide a constant table schema for querying structured logs
Gengliang Wang created SPARK-48123: -- Summary: Provide a constant table schema for querying structured logs Key: SPARK-48123 URL: https://issues.apache.org/jira/browse/SPARK-48123 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang Providing a table schema LOG_SCHEMA, so that users can load structured logs with the following: ``` spark.read.schema(LOG_SCHEMA).json(logPath) ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47671) Enable structured logging in log4j2.properties.template and update `configuration.md`
[ https://issues.apache.org/jira/browse/SPARK-47671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47671: --- Description: # rename the current log4j2.properties.template as log4j2.properties.pattern-layout-template # Enable structured logging in log4j2.properties.template # Update `configuration.md` on how to configure logging was: # rename the current log4j2.properties.template as log4j2-pattern-layout.properties.template # Enable structured logging in log4j2.properties.template # Update `configuration.md` on how to configure logging > Enable structured logging in log4j2.properties.template and update > `configuration.md` > - > > Key: SPARK-47671 > URL: https://issues.apache.org/jira/browse/SPARK-47671 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > > # rename the current log4j2.properties.template as > log4j2.properties.pattern-layout-template > # Enable structured logging in log4j2.properties.template > # Update `configuration.md` on how to configure logging -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48067) Fix Variant default columns for more complex default variants
[ https://issues.apache.org/jira/browse/SPARK-48067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48067. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46312 [https://github.com/apache/spark/pull/46312] > Fix Variant default columns for more complex default variants > - > > Key: SPARK-48067 > URL: https://issues.apache.org/jira/browse/SPARK-48067 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Richard Chen >Assignee: Richard Chen >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Default columns are stored as structfield metadata (string -> string) map. > This means the literal values are stored as strings. > However, the string representation of a variant is the JSONified string. > Thus, we need to wrap the string with `parse_json` to correctly use the > default values -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48067) Fix Variant default columns for more complex default variants
[ https://issues.apache.org/jira/browse/SPARK-48067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-48067: -- Assignee: Richard Chen > Fix Variant default columns for more complex default variants > - > > Key: SPARK-48067 > URL: https://issues.apache.org/jira/browse/SPARK-48067 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Richard Chen >Assignee: Richard Chen >Priority: Major > Labels: pull-request-available > > Default columns are stored as structfield metadata (string -> string) map. > This means the literal values are stored as strings. > However, the string representation of a variant is the JSONified string. > Thus, we need to wrap the string with `parse_json` to correctly use the > default values -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47585) SQL core: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47585. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46264 [https://github.com/apache/spark/pull/46264] > SQL core: Migrate logInfo with variables to structured logging framework > > > Key: SPARK-47585 > URL: https://issues.apache.org/jira/browse/SPARK-47585 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48016) Fix a bug in try_divide function when with decimals
[
https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-48016:
---
Fix Version/s: 4.0.0
3.5.2
> Fix a bug in try_divide function when with decimals
> ---
>
> Key: SPARK-48016
> URL: https://issues.apache.org/jira/browse/SPARK-48016
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0, 3.5.2
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.5.2, 3.4.4
>
>
> Binary Arithmetic operators should include the evalMode during makeCopy.
> Otherwise, the following query will throw DIVIDE_BY_ZERO error instead of
> returning null
>
> {code:java}
> SELECT try_divide(1, decimal(0)); {code}
> This is caused from the rule DecimalPrecision:
> {code:java}
> case b @ BinaryOperator(left, right) if left.dataType != right.dataType =>
> (left, right) match {
> ...
> case (l: Literal, r) if r.dataType.isInstanceOf[DecimalType] &&
> l.dataType.isInstanceOf[IntegralType] &&
> literalPickMinimumPrecision =>
> b.makeCopy(Array(Cast(l, DataTypeUtils.fromLiteral(l)), r)) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48016) Fix a bug in try_divide function when with decimals
[
https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang resolved SPARK-48016.
Fix Version/s: 3.4.4
Resolution: Fixed
Issue resolved by pull request 46289
[https://github.com/apache/spark/pull/46289]
> Fix a bug in try_divide function when with decimals
> ---
>
> Key: SPARK-48016
> URL: https://issues.apache.org/jira/browse/SPARK-48016
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0, 3.5.2
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.4
>
>
> Binary Arithmetic operators should include the evalMode during makeCopy.
> Otherwise, the following query will throw DIVIDE_BY_ZERO error instead of
> returning null
>
> {code:java}
> SELECT try_divide(1, decimal(0)); {code}
> This is caused from the rule DecimalPrecision:
> {code:java}
> case b @ BinaryOperator(left, right) if left.dataType != right.dataType =>
> (left, right) match {
> ...
> case (l: Literal, r) if r.dataType.isInstanceOf[DecimalType] &&
> l.dataType.isInstanceOf[IntegralType] &&
> literalPickMinimumPrecision =>
> b.makeCopy(Array(Cast(l, DataTypeUtils.fromLiteral(l)), r)) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48016) Fix a bug in try_divide function when with decimals
[
https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842193#comment-17842193
]
Gengliang Wang commented on SPARK-48016:
[~dongjoon] sure, I just moved it.
> Fix a bug in try_divide function when with decimals
> ---
>
> Key: SPARK-48016
> URL: https://issues.apache.org/jira/browse/SPARK-48016
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0, 3.5.2
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
> Labels: pull-request-available
>
> Binary Arithmetic operators should include the evalMode during makeCopy.
> Otherwise, the following query will throw DIVIDE_BY_ZERO error instead of
> returning null
>
> {code:java}
> SELECT try_divide(1, decimal(0)); {code}
> This is caused from the rule DecimalPrecision:
> {code:java}
> case b @ BinaryOperator(left, right) if left.dataType != right.dataType =>
> (left, right) match {
> ...
> case (l: Literal, r) if r.dataType.isInstanceOf[DecimalType] &&
> l.dataType.isInstanceOf[IntegralType] &&
> literalPickMinimumPrecision =>
> b.makeCopy(Array(Cast(l, DataTypeUtils.fromLiteral(l)), r)) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48016) Fix a bug in try_divide function when with decimals
[
https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-48016:
---
Parent Issue: SPARK-44111 (was: SPARK-35161)
> Fix a bug in try_divide function when with decimals
> ---
>
> Key: SPARK-48016
> URL: https://issues.apache.org/jira/browse/SPARK-48016
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0, 3.5.2
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
> Labels: pull-request-available
>
> Binary Arithmetic operators should include the evalMode during makeCopy.
> Otherwise, the following query will throw DIVIDE_BY_ZERO error instead of
> returning null
>
> {code:java}
> SELECT try_divide(1, decimal(0)); {code}
> This is caused from the rule DecimalPrecision:
> {code:java}
> case b @ BinaryOperator(left, right) if left.dataType != right.dataType =>
> (left, right) match {
> ...
> case (l: Literal, r) if r.dataType.isInstanceOf[DecimalType] &&
> l.dataType.isInstanceOf[IntegralType] &&
> literalPickMinimumPrecision =>
> b.makeCopy(Array(Cast(l, DataTypeUtils.fromLiteral(l)), r)) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48035) try_add/try_multiply should not be semantic equal to add/multiply
Gengliang Wang created SPARK-48035:
--
Summary: try_add/try_multiply should not be semantic equal to
add/multiply
Key: SPARK-48035
URL: https://issues.apache.org/jira/browse/SPARK-48035
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: Gengliang Wang
In the current implementation, the following code will return true
{code:java}
val l1 = Literal(1)
val l2 = Literal(2)
val l3 = Literal(3)
val expr1 = Add(Add(l1, l2), l3)
val expr2 = Add(Add(l2, l1, EvalMode.TRY), l3)
expr1.semanticEquals(expr2) {code}
The same applies to Multiply.
When creating MultiCommutativeOp for Add/Multiply, we should ensure all the
evalMode are consistent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48016) Binary Arithmetic operators should include the evalMode when makeCopy
[
https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-48016:
---
Summary: Binary Arithmetic operators should include the evalMode when
makeCopy (was: Binary Arithmetic operators should include the evalMode during
makeCopy)
> Binary Arithmetic operators should include the evalMode when makeCopy
> -
>
> Key: SPARK-48016
> URL: https://issues.apache.org/jira/browse/SPARK-48016
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0, 3.5.2
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
>
> Binary Arithmetic operators should include the evalMode during makeCopy.
> Otherwise, the following query will throw DIVIDE_BY_ZERO error instead of
> returning null
>
> {code:java}
> SELECT try_divide(1, decimal(0)); {code}
> This is caused from the rule DecimalPrecision:
> {code:java}
> case b @ BinaryOperator(left, right) if left.dataType != right.dataType =>
> (left, right) match {
> ...
> case (l: Literal, r) if r.dataType.isInstanceOf[DecimalType] &&
> l.dataType.isInstanceOf[IntegralType] &&
> literalPickMinimumPrecision =>
> b.makeCopy(Array(Cast(l, DataTypeUtils.fromLiteral(l)), r)) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48016) Binary Arithmetic operators should include the evalMode during makeCopy
Gengliang Wang created SPARK-48016:
--
Summary: Binary Arithmetic operators should include the evalMode
during makeCopy
Key: SPARK-48016
URL: https://issues.apache.org/jira/browse/SPARK-48016
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0, 3.5.2
Reporter: Gengliang Wang
Assignee: Gengliang Wang
Binary Arithmetic operators should include the evalMode during makeCopy.
Otherwise, the following query will throw DIVIDE_BY_ZERO error instead of
returning null
{code:java}
SELECT try_divide(1, decimal(0)); {code}
This is caused from the rule DecimalPrecision:
{code:java}
case b @ BinaryOperator(left, right) if left.dataType != right.dataType =>
(left, right) match {
...
case (l: Literal, r) if r.dataType.isInstanceOf[DecimalType] &&
l.dataType.isInstanceOf[IntegralType] &&
literalPickMinimumPrecision =>
b.makeCopy(Array(Cast(l, DataTypeUtils.fromLiteral(l)), r)) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47696) try_to_timestamp should handle SparkUpgradeException
[
https://issues.apache.org/jira/browse/SPARK-47696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang resolved SPARK-47696.
Resolution: Won't Fix
> try_to_timestamp should handle SparkUpgradeException
>
>
> Key: SPARK-47696
> URL: https://issues.apache.org/jira/browse/SPARK-47696
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0, 3.5.2, 3.4.3
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
> Labels: pull-request-available
>
> Currently, try_to_timestamp will throw an exception on legacy timestamp input.
> {code:java}
> > SELECT try_to_timestamp('2016-12-1', '-MM-dd')
> org.apache.spark.SparkUpgradeException:
> [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may
> get a different result due to the upgrading to Spark >= 3.0:
> Fail to parse '2016-12-1' in the new parser.
> You can set "spark.sql.legacy.timeParserPolicy" to "LEGACY" to restore the
> behavior before Spark 3.0, or set to "CORRECTED" and treat it as an invalid
> datetime string. SQLSTATE: 42K0B {code}
> It should return null instead of error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48011) Store LogKey name as a value to avoid generating new string instances
Gengliang Wang created SPARK-48011: -- Summary: Store LogKey name as a value to avoid generating new string instances Key: SPARK-48011 URL: https://issues.apache.org/jira/browse/SPARK-48011 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47963) Make the external Spark ecosystem can use structured logging mechanisms
[ https://issues.apache.org/jira/browse/SPARK-47963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47963. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46193 [https://github.com/apache/spark/pull/46193] > Make the external Spark ecosystem can use structured logging mechanisms > > > Key: SPARK-47963 > URL: https://issues.apache.org/jira/browse/SPARK-47963 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47583) SQL core: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47583. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45969 [https://github.com/apache/spark/pull/45969] > SQL core: Migrate logError with variables to structured logging framework > - > > Key: SPARK-47583 > URL: https://issues.apache.org/jira/browse/SPARK-47583 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Daniel >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47604) Resource managers: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47604. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46130 [https://github.com/apache/spark/pull/46130] > Resource managers: Migrate logInfo with variables to structured logging > framework > - > > Key: SPARK-47604 > URL: https://issues.apache.org/jira/browse/SPARK-47604 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47600) MLLib: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47600. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46151 [https://github.com/apache/spark/pull/46151] > MLLib: Migrate logInfo with variables to structured logging framework > - > > Key: SPARK-47600 > URL: https://issues.apache.org/jira/browse/SPARK-47600 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47907) Put removal of '!' as a synonym for 'NOT' on a keyword level under a config
[ https://issues.apache.org/jira/browse/SPARK-47907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47907: -- Assignee: Serge Rielau > Put removal of '!' as a synonym for 'NOT' on a keyword level under a config > --- > > Key: SPARK-47907 > URL: https://issues.apache.org/jira/browse/SPARK-47907 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > > Recently we dissolved the lexer equivalence between '!' and 'NOT'. > ! is a prefix operator and a synonym for NOT only in that case. > But NOT is used in many more cases in the grammar. > Given that there are a handful of known scenearios where users have exploited > the undocumented loophole it's best to add a config. > Usage found so far is: > `c1 ! IN(1, 2)` > `c1 ! BETWEEN 1 AND 2` > `c1 ! LIKE 'a%'` > But there are worse cases: > c1 IS ! NULL > CREATE TABLE T(c1 INT ! NULL) > or even > CREATE TABLE IF ! EXISTS T(c1 INT) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47907) Put removal of '!' as a synonym for 'NOT' on a keyword level under a config
[ https://issues.apache.org/jira/browse/SPARK-47907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47907. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46138 [https://github.com/apache/spark/pull/46138] > Put removal of '!' as a synonym for 'NOT' on a keyword level under a config > --- > > Key: SPARK-47907 > URL: https://issues.apache.org/jira/browse/SPARK-47907 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Recently we dissolved the lexer equivalence between '!' and 'NOT'. > ! is a prefix operator and a synonym for NOT only in that case. > But NOT is used in many more cases in the grammar. > Given that there are a handful of known scenearios where users have exploited > the undocumented loophole it's best to add a config. > Usage found so far is: > `c1 ! IN(1, 2)` > `c1 ! BETWEEN 1 AND 2` > `c1 ! LIKE 'a%'` > But there are worse cases: > c1 IS ! NULL > CREATE TABLE T(c1 INT ! NULL) > or even > CREATE TABLE IF ! EXISTS T(c1 INT) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47596) Streaming: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47596. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46079 [https://github.com/apache/spark/pull/46079] > Streaming: Migrate logWarn with variables to structured logging framework > - > > Key: SPARK-47596 > URL: https://issues.apache.org/jira/browse/SPARK-47596 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47591) Hive-thriftserver: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47591. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45926 [https://github.com/apache/spark/pull/45926] > Hive-thriftserver: Migrate logInfo with variables to structured logging > framework > - > > Key: SPARK-47591 > URL: https://issues.apache.org/jira/browse/SPARK-47591 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47584) SQL core: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47584. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46057 [https://github.com/apache/spark/pull/46057] > SQL core: Migrate logWarn with variables to structured logging framework > > > Key: SPARK-47584 > URL: https://issues.apache.org/jira/browse/SPARK-47584 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47627) MERGE with WITH SCHEMA EVOLUTION keywords
[ https://issues.apache.org/jira/browse/SPARK-47627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47627. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45748 [https://github.com/apache/spark/pull/45748] > MERGE with WITH SCHEMA EVOLUTION keywords > - > > Key: SPARK-47627 > URL: https://issues.apache.org/jira/browse/SPARK-47627 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 4.0.0 >Reporter: Pengfei Xu >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47588) Hive module: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47588. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46086 [https://github.com/apache/spark/pull/46086] > Hive module: Migrate logInfo with variables to structured logging framework > --- > > Key: SPARK-47588 > URL: https://issues.apache.org/jira/browse/SPARK-47588 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47590) Hive-thriftserver: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47590. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45923 [https://github.com/apache/spark/pull/45923] > Hive-thriftserver: Migrate logWarn with variables to structured logging > framework > - > > Key: SPARK-47590 > URL: https://issues.apache.org/jira/browse/SPARK-47590 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47594) Connector module: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47594: -- Assignee: BingKun Pan > Connector module: Migrate logInfo with variables to structured logging > framework > > > Key: SPARK-47594 > URL: https://issues.apache.org/jira/browse/SPARK-47594 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47594) Connector module: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47594. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46022 [https://github.com/apache/spark/pull/46022] > Connector module: Migrate logInfo with variables to structured logging > framework > > > Key: SPARK-47594 > URL: https://issues.apache.org/jira/browse/SPARK-47594 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47804) Add Dataframe cache debug log
[ https://issues.apache.org/jira/browse/SPARK-47804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47804. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45990 [https://github.com/apache/spark/pull/45990] > Add Dataframe cache debug log > - > > Key: SPARK-47804 > URL: https://issues.apache.org/jira/browse/SPARK-47804 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Xinyi Yu >Assignee: Xinyi Yu >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Add debug log for dataframe cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47603) Resource managers: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47603. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45957 [https://github.com/apache/spark/pull/45957] > Resource managers: Migrate logWarn with variables to structured logging > framework > - > > Key: SPARK-47603 > URL: https://issues.apache.org/jira/browse/SPARK-47603 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47792) Make the value of MDC can support `null`
[ https://issues.apache.org/jira/browse/SPARK-47792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47792. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45975 [https://github.com/apache/spark/pull/45975] > Make the value of MDC can support `null` > > > Key: SPARK-47792 > URL: https://issues.apache.org/jira/browse/SPARK-47792 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47587) Hive module: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47587. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45927 [https://github.com/apache/spark/pull/45927] > Hive module: Migrate logWarn with variables to structured logging framework > --- > > Key: SPARK-47587 > URL: https://issues.apache.org/jira/browse/SPARK-47587 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47601) Graphx: Migrate logs with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47601. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45947 [https://github.com/apache/spark/pull/45947] > Graphx: Migrate logs with variables to structured logging framework > > > Key: SPARK-47601 > URL: https://issues.apache.org/jira/browse/SPARK-47601 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47601) Graphx: Migrate logs with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47601: -- Assignee: Haejoon Lee > Graphx: Migrate logs with variables to structured logging framework > > > Key: SPARK-47601 > URL: https://issues.apache.org/jira/browse/SPARK-47601 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47595) Streaming: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47595. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45910 [https://github.com/apache/spark/pull/45910] > Streaming: Migrate logError with variables to structured logging framework > -- > > Key: SPARK-47595 > URL: https://issues.apache.org/jira/browse/SPARK-47595 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47593) Connector module: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47593. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45879 [https://github.com/apache/spark/pull/45879] > Connector module: Migrate logWarn with variables to structured logging > framework > > > Key: SPARK-47593 > URL: https://issues.apache.org/jira/browse/SPARK-47593 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47586) Hive module: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47586. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45876 [https://github.com/apache/spark/pull/45876] > Hive module: Migrate logError with variables to structured logging framework > > > Key: SPARK-47586 > URL: https://issues.apache.org/jira/browse/SPARK-47586 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47783) Refresh error-states.json
[ https://issues.apache.org/jira/browse/SPARK-47783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47783: -- Assignee: Serge Rielau > Refresh error-states.json > - > > Key: SPARK-47783 > URL: https://issues.apache.org/jira/browse/SPARK-47783 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > > We want to add more SQLSTATEs to the menu to prevent collisions and do some > general cleanup -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47783) Refresh error-states.json
[ https://issues.apache.org/jira/browse/SPARK-47783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47783. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45961 [https://github.com/apache/spark/pull/45961] > Refresh error-states.json > - > > Key: SPARK-47783 > URL: https://issues.apache.org/jira/browse/SPARK-47783 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > We want to add more SQLSTATEs to the menu to prevent collisions and do some > general cleanup -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47581) SQL catalyst: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47581. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45904 [https://github.com/apache/spark/pull/45904] > SQL catalyst: Migrate logWarn with variables to structured logging framework > > > Key: SPARK-47581 > URL: https://issues.apache.org/jira/browse/SPARK-47581 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47589) Hive-thriftserver: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47589. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45936 [https://github.com/apache/spark/pull/45936] > Hive-thriftserver: Migrate logError with variables to structured logging > framework > -- > > Key: SPARK-47589 > URL: https://issues.apache.org/jira/browse/SPARK-47589 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47588) Hive module: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835080#comment-17835080 ] Gengliang Wang commented on SPARK-47588: I am working on this one > Hive module: Migrate logInfo with variables to structured logging framework > --- > > Key: SPARK-47588 > URL: https://issues.apache.org/jira/browse/SPARK-47588 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47587) Hive module: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47587: -- Assignee: BingKun Pan > Hive module: Migrate logWarn with variables to structured logging framework > --- > > Key: SPARK-47587 > URL: https://issues.apache.org/jira/browse/SPARK-47587 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47586) Hive module: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47586: -- Assignee: Haejoon Lee > Hive module: Migrate logError with variables to structured logging framework > > > Key: SPARK-47586 > URL: https://issues.apache.org/jira/browse/SPARK-47586 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47591) Hive-thriftserver: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47591: -- Assignee: Haejoon Lee > Hive-thriftserver: Migrate logInfo with variables to structured logging > framework > - > > Key: SPARK-47591 > URL: https://issues.apache.org/jira/browse/SPARK-47591 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47592) Connector module: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47592. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45877 [https://github.com/apache/spark/pull/45877] > Connector module: Migrate logError with variables to structured logging > framework > - > > Key: SPARK-47592 > URL: https://issues.apache.org/jira/browse/SPARK-47592 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47743) Use milliseconds as the time unit in loggings
[ https://issues.apache.org/jira/browse/SPARK-47743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47743. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45903 [https://github.com/apache/spark/pull/45903] > Use milliseconds as the time unit in loggings > - > > Key: SPARK-47743 > URL: https://issues.apache.org/jira/browse/SPARK-47743 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47582) SQL catalyst: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47582. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45866 [https://github.com/apache/spark/pull/45866] > SQL catalyst: Migrate logInfo with variables to structured logging framework > > > Key: SPARK-47582 > URL: https://issues.apache.org/jira/browse/SPARK-47582 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Daniel >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47743) Use milliseconds as the time unit in loggings
Gengliang Wang created SPARK-47743: -- Summary: Use milliseconds as the time unit in loggings Key: SPARK-47743 URL: https://issues.apache.org/jira/browse/SPARK-47743 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47719) Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED
[ https://issues.apache.org/jira/browse/SPARK-47719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47719. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45859 [https://github.com/apache/spark/pull/45859] > Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to > CORRECTED > --- > > Key: SPARK-47719 > URL: https://issues.apache.org/jira/browse/SPARK-47719 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > spark.sql.legacy.timeParserPolicy was introduced in Spark 3.0 and has been > set to EXCEPTION. > Changing it from EXCEPTION for SPark 4.0 to CORRECTED will reduce errors and > reflects a prudent timeframe. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47719) Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED
[ https://issues.apache.org/jira/browse/SPARK-47719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47719: -- Assignee: Serge Rielau > Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to > CORRECTED > --- > > Key: SPARK-47719 > URL: https://issues.apache.org/jira/browse/SPARK-47719 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > > spark.sql.legacy.timeParserPolicy was introduced in Spark 3.0 and has been > set to EXCEPTION. > Changing it from EXCEPTION for SPark 4.0 to CORRECTED will reduce errors and > reflects a prudent timeframe. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47598) MLLib: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47598. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45837 [https://github.com/apache/spark/pull/45837] > MLLib: Migrate logError with variables to structured logging framework > -- > > Key: SPARK-47598 > URL: https://issues.apache.org/jira/browse/SPARK-47598 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47577) Spark core: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47577. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45834 [https://github.com/apache/spark/pull/45834] > Spark core: Migrate logError with variables to structured logging framework > --- > > Key: SPARK-47577 > URL: https://issues.apache.org/jira/browse/SPARK-47577 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47705) Sort LogKey alphabetically
[ https://issues.apache.org/jira/browse/SPARK-47705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47705. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45857 [https://github.com/apache/spark/pull/45857] > Sort LogKey alphabetically > -- > > Key: SPARK-47705 > URL: https://issues.apache.org/jira/browse/SPARK-47705 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Daniel >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Sort LogKey alphabetically and build a test to ensure it. > https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47705) Sort LogKey alphabetically
[ https://issues.apache.org/jira/browse/SPARK-47705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47705: -- Assignee: Daniel > Sort LogKey alphabetically > -- > > Key: SPARK-47705 > URL: https://issues.apache.org/jira/browse/SPARK-47705 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Daniel >Priority: Major > Labels: pull-request-available > > Sort LogKey alphabetically and build a test to ensure it. > https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47721) Guidelines for the Structured Logging Framework
Gengliang Wang created SPARK-47721: -- Summary: Guidelines for the Structured Logging Framework Key: SPARK-47721 URL: https://issues.apache.org/jira/browse/SPARK-47721 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47705) Sort LogKey alphabetically
Gengliang Wang created SPARK-47705: -- Summary: Sort LogKey alphabetically Key: SPARK-47705 URL: https://issues.apache.org/jira/browse/SPARK-47705 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Gengliang Wang Sort LogKey alphabetically and build a test to ensure it. https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47588) Hive module: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47588: --- Component/s: Spark Core (was: Project Infra) > Hive module: Migrate logInfo with variables to structured logging framework > --- > > Key: SPARK-47588 > URL: https://issues.apache.org/jira/browse/SPARK-47588 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47602) Resource managers: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47602: --- Component/s: Spark Core (was: Project Infra) > Resource managers: Migrate logError with variables to structured logging > framework > -- > > Key: SPARK-47602 > URL: https://issues.apache.org/jira/browse/SPARK-47602 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47583) SQL core: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47583: --- Component/s: Spark Core (was: Project Infra) > SQL core: Migrate logError with variables to structured logging framework > - > > Key: SPARK-47583 > URL: https://issues.apache.org/jira/browse/SPARK-47583 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47659) Improve `*LoggingSuite*`
[ https://issues.apache.org/jira/browse/SPARK-47659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47659: --- Component/s: (was: Tests) > Improve `*LoggingSuite*` > > > Key: SPARK-47659 > URL: https://issues.apache.org/jira/browse/SPARK-47659 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47589) Hive-thriftserver: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47589: --- Component/s: Spark Core (was: Project Infra) > Hive-thriftserver: Migrate logError with variables to structured logging > framework > -- > > Key: SPARK-47589 > URL: https://issues.apache.org/jira/browse/SPARK-47589 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47606) Create log4j templates for both structured logging and plain text logging
[ https://issues.apache.org/jira/browse/SPARK-47606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47606: --- Component/s: Spark Core (was: Project Infra) > Create log4j templates for both structured logging and plain text logging > - > > Key: SPARK-47606 > URL: https://issues.apache.org/jira/browse/SPARK-47606 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47598) MLLib: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47598: --- Component/s: Spark Core (was: Project Infra) > MLLib: Migrate logError with variables to structured logging framework > -- > > Key: SPARK-47598 > URL: https://issues.apache.org/jira/browse/SPARK-47598 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47600) MLLib: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47600: --- Component/s: Spark Core (was: Project Infra) > MLLib: Migrate logInfo with variables to structured logging framework > - > > Key: SPARK-47600 > URL: https://issues.apache.org/jira/browse/SPARK-47600 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47671) Enable structured logging in log4j2.properties.template and update `configuration.md`
[ https://issues.apache.org/jira/browse/SPARK-47671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47671: --- Component/s: Spark Core (was: Project Infra) > Enable structured logging in log4j2.properties.template and update > `configuration.md` > - > > Key: SPARK-47671 > URL: https://issues.apache.org/jira/browse/SPARK-47671 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > > # rename the current log4j2.properties.template as > log4j2-pattern-layout.properties.template > # Enable structured logging in log4j2.properties.template > # Update `configuration.md` on how to configure logging -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47581) SQL catalyst: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47581: --- Component/s: Spark Core (was: Project Infra) > SQL catalyst: Migrate logWarn with variables to structured logging framework > > > Key: SPARK-47581 > URL: https://issues.apache.org/jira/browse/SPARK-47581 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47593) Connector module: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47593: --- Component/s: Spark Core (was: Project Infra) > Connector module: Migrate logWarn with variables to structured logging > framework > > > Key: SPARK-47593 > URL: https://issues.apache.org/jira/browse/SPARK-47593 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47697) Add scala style check for invalid MDC usage
[
https://issues.apache.org/jira/browse/SPARK-47697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-47697:
---
Component/s: Spark Core
(was: Project Infra)
> Add scala style check for invalid MDC usage
> ---
>
> Key: SPARK-47697
> URL: https://issues.apache.org/jira/browse/SPARK-47697
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> To avoid invalid MDC usage s"Task ${MDC(TASK_ID, taskId)} failed", which
> should be log"Task ${MDC(TASK_ID, taskId)} failed. This makes development and
> PR review easier.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47601) Graphx: Migrate logs with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47601: --- Component/s: Spark Core (was: Project Infra) > Graphx: Migrate logs with variables to structured logging framework > > > Key: SPARK-47601 > URL: https://issues.apache.org/jira/browse/SPARK-47601 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47586) Hive module: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47586: --- Component/s: Spark Core (was: Project Infra) > Hive module: Migrate logError with variables to structured logging framework > > > Key: SPARK-47586 > URL: https://issues.apache.org/jira/browse/SPARK-47586 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47603) Resource managers: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47603: --- Component/s: Spark Core (was: Project Infra) > Resource managers: Migrate logWarn with variables to structured logging > framework > - > > Key: SPARK-47603 > URL: https://issues.apache.org/jira/browse/SPARK-47603 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47590) Hive-thriftserver: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47590: --- Component/s: Spark Core (was: Project Infra) > Hive-thriftserver: Migrate logWarn with variables to structured logging > framework > - > > Key: SPARK-47590 > URL: https://issues.apache.org/jira/browse/SPARK-47590 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47584) SQL core: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47584: --- Component/s: Spark Core (was: Project Infra) > SQL core: Migrate logWarn with variables to structured logging framework > > > Key: SPARK-47584 > URL: https://issues.apache.org/jira/browse/SPARK-47584 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47604) Resource managers: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47604: --- Component/s: Spark Core (was: Project Infra) > Resource managers: Migrate logInfo with variables to structured logging > framework > - > > Key: SPARK-47604 > URL: https://issues.apache.org/jira/browse/SPARK-47604 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47592) Connector module: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47592: --- Component/s: Spark Core (was: Project Infra) > Connector module: Migrate logError with variables to structured logging > framework > - > > Key: SPARK-47592 > URL: https://issues.apache.org/jira/browse/SPARK-47592 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47605) Enable structured logging in all the test log4j2.properties
[ https://issues.apache.org/jira/browse/SPARK-47605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47605: --- Component/s: Spark Core (was: Project Infra) > Enable structured logging in all the test log4j2.properties > --- > > Key: SPARK-47605 > URL: https://issues.apache.org/jira/browse/SPARK-47605 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47587) Hive module: Migrate logWarn with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47587: --- Component/s: Spark Core (was: Project Infra) > Hive module: Migrate logWarn with variables to structured logging framework > --- > > Key: SPARK-47587 > URL: https://issues.apache.org/jira/browse/SPARK-47587 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47574) Introduce Structured Logging Framework
[
https://issues.apache.org/jira/browse/SPARK-47574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-47574:
---
Component/s: Spark Core
(was: Project Infra)
> Introduce Structured Logging Framework
> --
>
> Key: SPARK-47574
> URL: https://issues.apache.org/jira/browse/SPARK-47574
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Introduce Structured Logging Framework as per
> [https://docs.google.com/document/d/1rATVGmFLNVLmtxSpWrEceYm7d-ocgu8ofhryVs4g3XU/edit?usp=sharing]
> .
> * The default logging output format will be json lines. For example
> {code:java}
> {
> "ts":"2023-03-12T12:02:46.661-0700",
> "level":"ERROR",
> "msg":"Cannot determine whether executor 289 is alive or not",
> "context":{
> "executor_id":"289"
> },
> "exception":{
> "class":"org.apache.spark.SparkException",
> "msg":"Exception thrown in awaitResult",
> "stackTrace":"..."
> },
> "source":"BlockManagerMasterEndpoint"
> } {code}
> * Introduce a new configuration `spark.log.structuredLogging.enabled` to
> control the default log4j configuration. Users can set it as false to get
> plain text log outputs
> * The change will start with logError method. Example changes on the API:
> from
> `logError(s"Cannot determine whether executor $executorId is alive or not.",
> e)` to `logError(log"Cannot determine whether executor ${MDC(EXECUTOR_ID,
> executorId)} is alive or not.", e)`
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47594) Connector module: Migrate logInfo with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47594: --- Component/s: Spark Core (was: Project Infra) > Connector module: Migrate logInfo with variables to structured logging > framework > > > Key: SPARK-47594 > URL: https://issues.apache.org/jira/browse/SPARK-47594 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47575) Implement logWarning API in structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47575: --- Component/s: Spark Core (was: Project Infra) > Implement logWarning API in structured logging framework > > > Key: SPARK-47575 > URL: https://issues.apache.org/jira/browse/SPARK-47575 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47576) Implement logInfo API in structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47576: --- Component/s: Spark Core (was: Project Infra) > Implement logInfo API in structured logging framework > - > > Key: SPARK-47576 > URL: https://issues.apache.org/jira/browse/SPARK-47576 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
