[
https://issues.apache.org/jira/browse/SPARK-49678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-49678:
----------------------------------
Description:
To allow users to control the default master setting during testing and
documentation generation.
*{*}BEFORE (`local[*]`){*}*
{code:java}
$ bin/pyspark
Python 3.9.19 (main, Jun 17 2024, 15:39:29)
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-pattern-layout-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
24/09/16 13:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ _/_ ___ ____/ /_
\ \/ _ \/ _ `/ __/ '/
/__ / ._/_,// //_\ version 4.0.0-SNAPSHOT
/_/
Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1726519982935).
SparkSession available as 'spark'.
>>>{code}
*{*}AFTER (`local[1]`){*}*
{code:java}
$ JDK_JAVA_OPTIONS="-Dspark.test.master=local[1]" bin/pyspark
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
Python 3.9.19 (main, Jun 17 2024, 15:39:29)
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-pattern-layout-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
24/09/16 13:51:03 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ _/_ ___ ____/ /_
\ \/ _ \/ _ `/ __/ '/
/__ / ._/_,// //_\ version 4.0.0-SNAPSHOT
/_/
Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[1], app id =
local-1726519863363).
SparkSession available as 'spark'.
>>>{code}
was:
To allow users to control the default master setting during testing and
documentation generation.
**BEFORE (`local[*]`)**
```
$ bin/pyspark
Python 3.9.19 (main, Jun 17 2024, 15:39:29)
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-pattern-layout-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
24/09/16 13:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1726519982935).
SparkSession available as 'spark'.
>>>
```
**AFTER (`local[1]`)**
```
$ JDK_JAVA_OPTIONS="-Dspark.test.master=local[1]" bin/pyspark
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
Python 3.9.19 (main, Jun 17 2024, 15:39:29)
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-pattern-layout-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
24/09/16 13:51:03 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[1], app id =
local-1726519863363).
SparkSession available as 'spark'.
>>>
```
> Support `spark.test.master` in `SparkSubmitArguments`
> -----------------------------------------------------
>
> Key: SPARK-49678
> URL: https://issues.apache.org/jira/browse/SPARK-49678
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Dongjoon Hyun
> Priority: Major
> Labels: pull-request-available
>
> To allow users to control the default master setting during testing and
> documentation generation.
> *{*}BEFORE (`local[*]`){*}*
> {code:java}
> $ bin/pyspark
> Python 3.9.19 (main, Jun 17 2024, 15:39:29)
> [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> WARNING: Using incubator modules: jdk.incubator.vector
> Using Spark's default log4j profile:
> org/apache/spark/log4j2-pattern-layout-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 24/09/16 13:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Welcome to
> ____ __
> / _/_ ___ ____/ /_
> \ \/ _ \/ _ `/ __/ '/
> /__ / ._/_,// //_\ version 4.0.0-SNAPSHOT
> /_/
> Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
> Spark context Web UI available at http://localhost:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1726519982935).
> SparkSession available as 'spark'.
> >>>{code}
>
> *{*}AFTER (`local[1]`){*}*
> {code:java}
> $ JDK_JAVA_OPTIONS="-Dspark.test.master=local[1]" bin/pyspark
> NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
> Python 3.9.19 (main, Jun 17 2024, 15:39:29)
> [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
> NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
> WARNING: Using incubator modules: jdk.incubator.vector
> Using Spark's default log4j profile:
> org/apache/spark/log4j2-pattern-layout-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 24/09/16 13:51:03 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Welcome to
> ____ __
> / _/_ ___ ____/ /_
> \ \/ _ \/ _ `/ __/ '/
> /__ / ._/_,// //_\ version 4.0.0-SNAPSHOT
> /_/
> Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
> Spark context Web UI available at http://localhost:4040
> Spark context available as 'sc' (master = local[1], app id =
> local-1726519863363).
> SparkSession available as 'spark'.
> >>>{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
