[
https://issues.apache.org/jira/browse/SPARK-49197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-49197:
----------------------------------
Description:
When `launcher` module shows `Spark Command`, there is no redaction. Although
Spark Cluster is supposed to be in a secure environment, this could be
collected by a centralized log system. We need to do a proper redaction like we
do for `Launch Command:` output.
{code:java}
$ SPARK_NO_DAEMONIZE=1 SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true
-Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter
-Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4="
sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to
/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/logs/spark-dongjoon-org.apache.spark.deploy.master.Master-1-M3-Max.local.out
Spark Command: /Users/dongjoon/.jenv/versions/17/bin/java -cp
/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/conf/:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/slf4j-api-2.0.13.jar:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/*
-Dspark.master.rest.enabled=true
-Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter
-Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4=
-Xmx1g org.apache.spark.deploy.master.Master --host M3-Max.local --port 7077
--webui-port 8080
{code}
was:
When `launcher` module shows `Spark Command`, there is no redaction. Although
Spark Cluster is supposed to be in a secure environment, this could be
collected by a centralized log system. We need to do a proper redaction.
{code}
$ SPARK_NO_DAEMONIZE=1 SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true
-Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter
-Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4="
sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to
/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/logs/spark-dongjoon-org.apache.spark.deploy.master.Master-1-M3-Max.local.out
Spark Command: /Users/dongjoon/.jenv/versions/17/bin/java -cp
/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/conf/:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/slf4j-api-2.0.13.jar:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/*
-Dspark.master.rest.enabled=true
-Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter
-Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4=
-Xmx1g org.apache.spark.deploy.master.Master --host M3-Max.local --port 7077
--webui-port 8080
{code}
> Redact `Spark Command` output in `launcher` module
> --------------------------------------------------
>
> Key: SPARK-49197
> URL: https://issues.apache.org/jira/browse/SPARK-49197
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.0, 3.5.2, 3.4.3
> Reporter: Dongjoon Hyun
> Assignee: Dongjoon Hyun
> Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0, 3.4.4, 3.5.3
>
>
> When `launcher` module shows `Spark Command`, there is no redaction. Although
> Spark Cluster is supposed to be in a secure environment, this could be
> collected by a centralized log system. We need to do a proper redaction like
> we do for `Launch Command:` output.
> {code:java}
> $ SPARK_NO_DAEMONIZE=1 SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true
> -Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter
> -Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4="
> sbin/start-master.sh
> starting org.apache.spark.deploy.master.Master, logging to
> /Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/logs/spark-dongjoon-org.apache.spark.deploy.master.Master-1-M3-Max.local.out
> Spark Command: /Users/dongjoon/.jenv/versions/17/bin/java -cp
> /Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/conf/:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/slf4j-api-2.0.13.jar:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/*
> -Dspark.master.rest.enabled=true
> -Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter
> -Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4=
> -Xmx1g org.apache.spark.deploy.master.Master --host M3-Max.local --port 7077
> --webui-port 8080
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
