[GitHub] spark issue #21502: [SPARK-22575][SQL] Add destroy to Dataset

2018-10-28 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21502
  
can this be merged?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-10-27 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21588
  
ship it!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-10-12 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21588
  
@jerryshao @vanzin can this be merged to master?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22219: [SPARK-25224][SQL] Improvement of Spark SQL ThriftServer...

2018-10-09 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/22219
  
@HyukjinKwon  can this be merged?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22347: [SPARK-25353][SQL] executeTake in SparkPlan is modified ...

2018-10-09 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/22347
  
@dongjoon-hyun can this be merged?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-09-27 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21588
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore

2018-09-19 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21404
  
@dongjoon-hyun I was planning to do my own custom build by cherrypicking 
your PR if you had it available


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore

2018-09-19 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21404
  
bump


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore

2018-09-05 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21404
  
Also, can hive 3.1 be supported easily or are there some breaking changes? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore

2018-09-05 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21404
  
@dongjoon-hyun @wangyum @gatorsmile @HyukjinKwon Anything left on this? can 
it be merged to master?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key pa...

2018-07-17 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-17 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r202990383
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
 ---
@@ -99,8 +99,10 @@ private[spark] class StandaloneSchedulerBackend(
 // Start executors with a few necessary configs for registering with 
the scheduler
 val sparkJavaOpts = Utils.sparkJavaOpts(conf, 
SparkConf.isExecutorStartupConf)
 val javaOpts = sparkJavaOpts ++ extraJavaOpts
-val command = 
Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
-  args, sc.executorEnvs, classPathEntries ++ testingClassPath, 
libraryPathEntries, javaOpts)
+val javaOptsFiltered = javaOpts.filterNot { opt =>
+opt.startsWith("-Dspark.ssl.keyStorePassword") || 
opt.startsWith("-Dspark.ssl.keyPassword")}
--- End diff --

done


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-17 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r202990353
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala 
---
@@ -130,7 +130,9 @@ private[deploy] class Master(
 logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
 webUi = new MasterWebUI(this, webUiPort)
 webUi.bind()
-masterWebUiUrl = "http://; + masterPublicAddress + ":" + 
webUi.boundPort
+val SSL_ENABLED = conf.getBoolean("spark.ssl.enabled", false)
--- End diff --

done


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-17 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r202962837
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
 ---
@@ -100,7 +100,9 @@ private[spark] class StandaloneSchedulerBackend(
 val sparkJavaOpts = Utils.sparkJavaOpts(conf, 
SparkConf.isExecutorStartupConf)
 val javaOpts = sparkJavaOpts ++ extraJavaOpts
 val command = 
Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
-  args, sc.executorEnvs, classPathEntries ++ testingClassPath, 
libraryPathEntries, javaOpts)
+  args, sc.executorEnvs, classPathEntries ++ testingClassPath, 
libraryPathEntries
--- End diff --

done


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-17 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r202962897
  
--- Diff: 
core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala ---
@@ -282,6 +282,40 @@ class MasterSuite extends SparkFunSuite
 }
   }
 
+  test("SPARK-24621 https urls when ssl enabled") {
--- End diff --

done


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-17 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r202962872
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
 ---
@@ -100,7 +100,9 @@ private[spark] class StandaloneSchedulerBackend(
 val sparkJavaOpts = Utils.sparkJavaOpts(conf, 
SparkConf.isExecutorStartupConf)
 val javaOpts = sparkJavaOpts ++ extraJavaOpts
 val command = 
Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
-  args, sc.executorEnvs, classPathEntries ++ testingClassPath, 
libraryPathEntries, javaOpts)
+  args, sc.executorEnvs, classPathEntries ++ testingClassPath, 
libraryPathEntries
+  , javaOpts.filterNot { opt =>
--- End diff --

done


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key pa...

2018-07-17 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key pa...

2018-07-16 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
@mgaido91 - do you know what caused this?
Scalastyle checks failed at following occurrences:
[error] 
/home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala:
 Expected token RPAREN but got Token(RBRACE,},10480,})


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key pa...

2018-07-16 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key pa...

2018-07-16 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
@HyukjinKwon can u commit this? i fixed the Scala style


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide key pa...

2018-07-16 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-01 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r199345049
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala 
---
@@ -130,7 +130,13 @@ private[deploy] class Master(
 logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
 webUi = new MasterWebUI(this, webUiPort)
 webUi.bind()
-masterWebUiUrl = "http://; + masterPublicAddress + ":" + 
webUi.boundPort
+val SSL_ENABLED = conf.getBoolean("spark.ssl.enabled", false)
+val uriScheme = "http://;
+if (SSL_ENABLED) {
+  uriScheme = "https://;
--- End diff --

woops, i just updated val to var


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-01 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r199345033
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala 
---
@@ -130,7 +130,13 @@ private[deploy] class Master(
 logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
 webUi = new MasterWebUI(this, webUiPort)
 webUi.bind()
-masterWebUiUrl = "http://; + masterPublicAddress + ":" + 
webUi.boundPort
+val SSL_ENABLED = conf.getBoolean("spark.ssl.enabled", false)
+val uriScheme = "http://;
+if (SSL_ENABLED) {
+  uriScheme = "https://;
+}
+masterWebUiUrl = uriScheme + masterPublicAddress + ":" + 
webUi.boundPort
+//masterWebUiUrl = "http://; + masterPublicAddress + ":" + 
webUi.boundPort
--- End diff --

update in latest commit


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [SPARK-24621] [Core] [WebUI] - hide...

2018-07-01 Thread tooptoop4 
Github user tooptoop4 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21514#discussion_r199345024
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
 ---
@@ -100,7 +100,7 @@ private[spark] class StandaloneSchedulerBackend(
 val sparkJavaOpts = Utils.sparkJavaOpts(conf, 
SparkConf.isExecutorStartupConf)
 val javaOpts = sparkJavaOpts ++ extraJavaOpts
 val command = 
Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
-  args, sc.executorEnvs, classPathEntries ++ testingClassPath, 
libraryPathEntries, javaOpts)
+  args, sc.executorEnvs, classPathEntries ++ testingClassPath, 
libraryPathEntries, 
javaOpts.filterNot(_.startsWith("-Dspark.ssl.keyStorePassword")).filterNot(_.startsWith("-Dspark.ssl.keyPassword")))
--- End diff --

does your PR still end up having the literal 'storePassword' in ps output?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [Core] - hide key password from linux ps l...

2018-06-08 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
@dbtsai 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21514: [SPARK-22860] [Core] - hide key password from linux ps l...

2018-06-08 Thread tooptoop4 
Github user tooptoop4 commented on the issue:

https://github.com/apache/spark/pull/21514
  
I just went through the process of getting 'security approval' in my 
organisation to use Spark in Production. They ran a Qualsys Vulnerability 
Assessment scanner that among other things picked up a vulnerability called 
'presence of string keyStorePassword in process list' so the scan would still 
fail if the string keyStorePassword appeared at all.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21514: [SPARK-22860] [Core] - hide key password from lin...

2018-06-08 Thread tooptoop4 
GitHub user tooptoop4 opened a pull request:

https://github.com/apache/spark/pull/21514

[SPARK-22860] [Core] - hide key password from linux ps listing

## What changes were proposed in this pull request?
 
hide password from 'ps' linux command

## How was this patch tested?

Existing tests

Please review http://spark.apache.org/contributing.html before opening a 
pull request.
I confirm this is my own work

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tooptoop4/spark spark22860

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21514


commit 380d97d84dda6617f531e29f1d477e2360d29dfd
Author: Toop 
Date:   2018-06-08T23:26:02Z

[SPARK-22860] [Core] - hide key password from linux ps listing




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org