[GitHub] spark issue #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compression.co...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19218
  
What are multiple expressions?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compression.co...

[fjh100456]

Github user fjh100456 commented on the issue:

https://github.com/apache/spark/pull/19218
  
@gatorsmile 
I'd test manully. When table-level compression not configured, it always 
take the session level compression, and ignore the existing file compression. 
Seems like a bug, however, table files with multiple compressions do not affect 
the reading and writing. 
Is it ok to add a test to check reading and writing when there are multiple 
conpressions in the existing table files?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19984: [SPARK-22789] Map-only continuous processing exec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19984


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19984
  
Thanks! Merging to master!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19984
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19984
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85332/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19984
  
**[Test build #85332 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85332/testReport)**
 for PR 19984 at commit 
[`b4f7976`](https://github.com/apache/spark/commit/b4f79762c083735011bf98250c39c263876c8cc8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20034: [SPARK-22846][SQL] Fix table owner is null when c...

[BruceXu1991]

Github user BruceXu1991 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20034#discussion_r158577749
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
---
@@ -186,7 +186,7 @@ private[hive] class HiveClientImpl(
   /** Returns the configuration for the current session. */
   def conf: HiveConf = state.getConf
 
-  private val userName = state.getAuthenticator.getUserName
+  private val userName = conf.getUser
--- End diff --

well, if using spark 2.2.1's current implementation
```
private val userName = state.getAuthenticator.getUserName
```
when the implementation of state.getAuthenticator is 
**HadoopDefaultAuthenticator**, which is default in hive conf, the username is 
got. 

however, in the case that the implementation of state.getAuthenticator is 
**SessionStateUserAuthenticator**, which is used in my case, then username will 
be null.

the simplified code below explains the reason:
1) HadoopDefaultAuthenticator
```
public class HadoopDefaultAuthenticator implements 
HiveAuthenticationProvider {
@Override
  public String getUserName() {
return userName;
  }

  @Override
  public void setConf(Configuration conf) {
this.conf = conf;
UserGroupInformation ugi = null;
try {
  ugi = Utils.getUGI();
} catch (Exception e) {
  throw new RuntimeException(e);
}
this.userName = ugi.getShortUserName();
if (ugi.getGroupNames() != null) {
  this.groupNames = Arrays.asList(ugi.getGroupNames());
}
  }
}

public class Utils {
  public static UserGroupInformation getUGI() throws LoginException, 
IOException {
String doAs = System.getenv("HADOOP_USER_NAME");
if(doAs != null && doAs.length() > 0) {
  return UserGroupInformation.createProxyUser(doAs, 
UserGroupInformation.getLoginUser());
}
return UserGroupInformation.getCurrentUser();
  }
}
```
it shows that HadoopDefaultAuthenticator will get username through 
Utils.getUGI(), so the username is HADOOP_USER_NAME of LoginUser.

2)  SessionStateUserAuthenticator
```
public class SessionStateUserAuthenticator implements 
HiveAuthenticationProvider {
  @Override
  public void setConf(Configuration arg0) {
  }

  @Override
  public String getUserName() {
return sessionState.getUserName();
  }
}
```
it shows that SessionStateUserAuthenticator get the username through 
sessionState.getUserName(), which is null. Here is the [instantiation of 
SessionState in 
HiveClientImpl](https://github.com/apache/spark/blob/1cf3e3a26961d306eb17b7629d8742a4df45f339/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L187)
 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85331/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20059
  
**[Test build #85331 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85331/testReport)**
 for PR 20059 at commit 
[`f23bf0f`](https://github.com/apache/spark/commit/f23bf0fdf21f21224895a5c35e0d95956a29abf9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20061: [SPARK-22890][TEST] Basic tests for DateTimeOperations

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20061
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20061: [SPARK-22890][TEST] Basic tests for DateTimeOperations

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20061
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85330/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20061: [SPARK-22890][TEST] Basic tests for DateTimeOperations

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20061
  
**[Test build #85330 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85330/testReport)**
 for PR 20061 at commit 
[`e8e4d11`](https://github.com/apache/spark/commit/e8e4d11a504c4169848baeabbec84af2a1b3e6a8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/19977
  
might have...I'll check the performance.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20004: [Spark-22818][SQL] csv escape of quote escape

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/20004#discussion_r158576836
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
 ---
@@ -148,6 +149,9 @@ class CSVOptions(
 format.setDelimiter(delimiter)
 format.setQuote(quote)
 format.setQuoteEscape(escape)
+if (charToEscapeQuoteEscaping != '\u') {
--- End diff --

Because we always call `format.setQuoteEscape(escape)`, this default value 
becomes wrong. That means, if users set `\u`, the actual quoteEscape in 
`univocity` is `\`. 

Let us use `Option[Char]` as the type of `charToEscapeQuoteEscaping `? We 
call `format.setCharToEscapeQuoteEscaping ` only if it is defined. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20004: [Spark-22818][SQL] csv escape of quote escape

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20004
  
Thank you for your investigation! 

Basically, because the option `escape` is set to `\`, the default value of 
`charToEscapeQuoteEscaping` is actually `\` in effect.

Could you update the doc?



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20057: [SPARK-22880][SQL] Add cascadeTruncate option to JDBC da...

[danielvdende]

Github user danielvdende commented on the issue:

https://github.com/apache/spark/pull/20057
  
@gatorsmile could you explain why you have doubts about the feature? Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19977
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85329/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19977
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85329 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85329/testReport)**
 for PR 19977 at commit 
[`3ec440d`](https://github.com/apache/spark/commit/3ec440de914d8f71c35f30cdccec815b199b5f17).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20023: [SPARK-22036][SQL] Decimal multiplication with high prec...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20023
  
```
db2 => create table decimals_test(id int, a decimal(31,18), b 
decimal(31,18))
DB2I  The SQL command completed successfully.
db2 => insert into decimals_test values (1, 2.33, 1.123456789123456789)
DB2I  The SQL command completed successfully.
db2 => select a * b from decimals_test

1
-
SQL0802N  Arithmetic overflow or other arithmetic exception occurred.  
SQLSTATE=22003
```

I might not get your point. Above is the result I got. This is your 
scenario 3 or 2?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...

[timout]

Github user timout commented on the issue:

https://github.com/apache/spark/pull/17619
  
That does exactly what is supposed to do. And you absolutely right it 
related to executors.
I am sorry if it is not clear from my previous explanations.
Let us say:
Spark Streaming App - very long running app:
 Driver, started by marathon using  docker image, schedules (in mesos 
meaning) executors using 
 docker images.(net=HOST) (every executor started from docker image on some 
mesos agent)
So if some recoverable error happens, for instance: 
ExecutorLostFailure (executor 40 exited caused by one of the running tasks) 
Reason: Remote RPC client disassociated...(I do not know how about others but 
it is relatively often in my env.)
As result the executor will be dead and after 2 failures mesos agent node 
will be included in MesosCoarseGrainedSchedulerBackend black list and driver 
will never schedule (in mesos meaning) executor on it. So the app will 
starve... and notice will not die.
That exactly what happened with my streams apps before that patch.

That patch may be incompatible with master already but i can fix it if 
needed.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19984
  
**[Test build #85332 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85332/testReport)**
 for PR 19984 at commit 
[`b4f7976`](https://github.com/apache/spark/commit/b4f79762c083735011bf98250c39c263876c8cc8).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19954
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85327/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19954
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19954
  
**[Test build #85327 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85327/testReport)**
 for PR 19954 at commit 
[`f82c568`](https://github.com/apache/spark/commit/f82c5682565bc0e1bc34ec428faedb53ee5ddecd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20011
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20011
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85326/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20011
  
**[Test build #85326 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85326/testReport)**
 for PR 20011 at commit 
[`931b2d2`](https://github.com/apache/spark/commit/931b2d262aa02880631ca4c693a84fa4c4d12318).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20057: [SPARK-22880][SQL] Add cascadeTruncate option to JDBC da...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20057
  
Thank you for your contribution! I am doubting the value of this feature.

If you are interested in the JDBC-related work, could you take 
https://issues.apache.org/jira/browse/SPARK-22731? 

Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20059
  
**[Test build #85331 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85331/testReport)**
 for PR 20059 at commit 
[`f23bf0f`](https://github.com/apache/spark/commit/f23bf0fdf21f21224895a5c35e0d95956a29abf9).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19977
  
Yes; otherwise, it will introduce a performance regression, right?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compres...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19218#discussion_r158575144
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala
 ---
@@ -42,8 +43,15 @@ private[parquet] class ParquetOptions(
* Acceptable values are defined in 
[[shortParquetCompressionCodecNames]].
*/
   val compressionCodecClassName: String = {
-val codecName = parameters.getOrElse("compression",
-  sqlConf.parquetCompressionCodec).toLowerCase(Locale.ROOT)
+// `compression`, `parquet.compression`(i.e., 
ParquetOutputFormat.COMPRESSION), and
+// `spark.sql.parquet.compression.codec`
+// are in order of precedence from highest to lowest.
+val parquetCompressionConf = 
parameters.get(ParquetOutputFormat.COMPRESSION)
+val codecName = parameters
+  .get("compression")
+  .orElse(parquetCompressionConf)
--- End diff --

Yeah, we can submit a separate PR for that issue. The behavior change needs 
to be documented in SparkSQL doc.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20059: [SPARK-22648][K8s] Add documentation covering ini...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158575137
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -528,51 +579,90 @@ specific to Spark on Kubernetes.
   
 
 
-   spark.kubernetes.driver.limit.cores
-   (none)
-   
- Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for the driver pod.
-   
- 
- 
-   spark.kubernetes.executor.limit.cores
-   (none)
-   
- Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for each executor pod launched for the Spark Application.
-   
- 
- 
-   spark.kubernetes.node.selector.[labelKey]
-   (none)
-   
- Adds to the node selector of the driver pod and executor pods, with 
key labelKey and the value as the
- configuration's value. For example, setting 
spark.kubernetes.node.selector.identifier to 
myIdentifier
- will result in the driver pod and executors having a node selector 
with key identifier and value
-  myIdentifier. Multiple node selector keys can be added 
by setting multiple configurations with this prefix.
-
-  
- 
-   
spark.kubernetes.driverEnv.[EnvironmentVariableName]
-   (none)
-   
- Add the environment variable specified by 
EnvironmentVariableName to
- the Driver process. The user can specify multiple of these to set 
multiple environment variables.
-   
- 
-  
-
spark.kubernetes.mountDependencies.jarsDownloadDir
-/var/spark-data/spark-jars
-
-  Location to download jars to in the driver and executors.
-  This directory must be empty and will be mounted as an empty 
directory volume on the driver and executor pods.
-
-  
-   
- 
spark.kubernetes.mountDependencies.filesDownloadDir
- /var/spark-data/spark-files
- 
-   Location to download jars to in the driver and executors.
-   This directory must be empty and will be mounted as an empty 
directory volume on the driver and executor pods.
- 
-   
+  spark.kubernetes.driver.limit.cores
+  (none)
+  
+Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for the driver pod.
+  
+
+
+  spark.kubernetes.executor.limit.cores
+  (none)
+  
+Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for each executor pod launched for the Spark Application.
+  
+
+
+  spark.kubernetes.node.selector.[labelKey]
+  (none)
+  
+Adds to the node selector of the driver pod and executor pods, with 
key labelKey and the value as the
+configuration's value. For example, setting 
spark.kubernetes.node.selector.identifier to 
myIdentifier
+will result in the driver pod and executors having a node selector 
with key identifier and value
+ myIdentifier. Multiple node selector keys can be added 
by setting multiple configurations with this prefix.
+  
+
+
+  
spark.kubernetes.driverEnv.[EnvironmentVariableName]
+  (none)
+  
+Add the environment variable specified by 
EnvironmentVariableName to
+the Driver process. The user can specify multiple of these to set 
multiple environment variables.
+  
+
+
+  spark.kubernetes.mountDependencies.jarsDownloadDir
+  /var/spark-data/spark-jars
+  
+Location to download jars to in the driver and executors.
+This directory must be empty and will be mounted as an empty directory 
volume on the driver and executor pods.
+  
+
+
+  spark.kubernetes.mountDependencies.filesDownloadDir
+  /var/spark-data/spark-files
+  
+Location to download jars to in the driver and executors.
+This directory must be empty and will be mounted as an empty directory 
volume on the driver and executor pods.
+  
+
+
+  spark.kubernetes.mountDependencies.timeout
+  5 minutes
+  
+   Timeout before aborting the attempt to download and unpack dependencies 
from remote locations into the driver and executor pods.
+  
+
+
+  
spark.kubernetes.mountDependencies.maxThreadPoolSize
+  5
+  
+   Maximum size of the thread pool for downloading remote dependencies 
into the driver and executor pods.
--- End diff --

Done.


---

-
To unsubscribe, 

[GitHub] spark pull request #20059: [SPARK-22648][K8s] Add documentation covering ini...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158575135
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,57 @@ by their appropriate remote URIs. Also, application 
dependencies can be pre-moun
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
 `SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
 
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like 
HDFS or HTTP servers, the driver and executor pods
+need a Kubernetes 
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
 for downloading
+the dependencies so the driver and executor containers can use them 
locally. This requires users to specify the container
+image for the init-container using the configuration property 
`spark.kubernetes.initContainer.image`. For example, users
+simply add the following option to the `spark-submit` command to specify 
the init-container image:
+
+```
+--conf spark.kubernetes.initContainer.image=
+```
+
+The init-container handles remote dependencies specified in `spark.jars` 
(or the `--jars` option of `spark-submit`) and
+`spark.files` (or the `--files` option of `spark-submit`). It also handles 
remotely hosted main application resources, e.g.,
+the main application jar. The following shows an example of using remote 
dependencies with the `spark-submit` command:
+
+```bash
+$ bin/spark-submit \
+--master k8s://https://: \
+--deploy-mode cluster \
+--name spark-pi \
+--class org.apache.spark.examples.SparkPi \
+--jars https://path/to/dependency1.jar,https://path/to/dependency2.jar
+--files hdfs://host:port/path/to/file1,hdfs://host:port/path/to/file2
+--conf spark.executor.instances=5 \
+--conf spark.kubernetes.driver.docker.image= \
+--conf spark.kubernetes.executor.docker.image= \
+--conf spark.kubernetes.initContainer.image=
+https://path/to/examples.jar
+```
+
+## Secret Management
+In some cases, a Spark application may need to use some credentials, e.g., 
for accessing data on a secured HDFS cluster
--- End diff --

Done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20061: [SPARK-22890][TEST] Basic tests for DateTimeOperations

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20061
  
**[Test build #85330 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85330/testReport)**
 for PR 20061 at commit 
[`e8e4d11`](https://github.com/apache/spark/commit/e8e4d11a504c4169848baeabbec84af2a1b3e6a8).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20061: [SPARK-22890][TEST] Basic tests for DateTimeOpera...

[wangyum]

GitHub user wangyum opened a pull request:

https://github.com/apache/spark/pull/20061

[SPARK-22890][TEST] Basic tests for DateTimeOperations

## What changes were proposed in this pull request?

Test Coverage for `DateTimeOperations`, this is a Sub-tasks for 
[SPARK-22722](https://issues.apache.org/jira/browse/SPARK-22722).

## How was this patch tested?

N/A

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyum/spark SPARK-22890

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20061.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20061


commit 24b50f0c8371af258ed152363a9ba8148b23d2d2
Author: Yuming Wang 
Date:   2017-12-23T02:39:39Z

Basic tests for DateTimeOperations

commit e8e4d11a504c4169848baeabbec84af2a1b3e6a8
Author: Yuming Wang 
Date:   2017-12-23T02:53:40Z

Append a blank line




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compres...

[fjh100456]

Github user fjh100456 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19218#discussion_r158574986
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala
 ---
@@ -42,8 +43,15 @@ private[parquet] class ParquetOptions(
* Acceptable values are defined in 
[[shortParquetCompressionCodecNames]].
*/
   val compressionCodecClassName: String = {
-val codecName = parameters.getOrElse("compression",
-  sqlConf.parquetCompressionCodec).toLowerCase(Locale.ROOT)
+// `compression`, `parquet.compression`(i.e., 
ParquetOutputFormat.COMPRESSION), and
+// `spark.sql.parquet.compression.codec`
+// are in order of precedence from highest to lowest.
+val parquetCompressionConf = 
parameters.get(ParquetOutputFormat.COMPRESSION)
+val codecName = parameters
+  .get("compression")
+  .orElse(parquetCompressionConf)
--- End diff --

If so, parquet's table-level compression may be overwrited in this PR, and 
it may not be what we want.
Shall I  fix it first in another PR?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85325/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20059
  
**[Test build #85325 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85325/testReport)**
 for PR 20059 at commit 
[`fbb2112`](https://github.com/apache/spark/commit/fbb21121447fe131358042ce05454f88d6fb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85323/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20059
  
**[Test build #85323 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85323/testReport)**
 for PR 20059 at commit 
[`a6a6660`](https://github.com/apache/spark/commit/a6a666060787cadd832b5bd32281940a6c81a9a9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19954
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19954
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85324/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19954
  
**[Test build #85324 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85324/testReport)**
 for PR 19954 at commit 
[`785b90e`](https://github.com/apache/spark/commit/785b90e52580e9175896b22b00b23f30fbe020ef).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

[Ngone51]

Github user Ngone51 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20039#discussion_r158574199
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -125,13 +128,39 @@ private[spark] class LiveListenerBus(conf: SparkConf) 
{
 
   /** Post an event to all queues. */
   def post(event: SparkListenerEvent): Unit = {
-if (!stopped.get()) {
-  metrics.numEventsPosted.inc()
-  val it = queues.iterator()
-  while (it.hasNext()) {
-it.next().post(event)
+if (stopped.get()) {
+  return
+}
+
+metrics.numEventsPosted.inc()
+
+// If the event buffer is null, it means the bus has been started and 
we can avoid
+// synchronization and post events directly to the queues. This should 
be the most
+// common case during the life of the bus.
+if (queuedEvents == null) {
+  postToQueues(event)
--- End diff --

What if stop() called after the null judge and  before postToQueues() call 
? 
Do you think we should check the stopped.get() in postToQueues()?
like:
```
private def postToQueues(event: SparkListenerEvent): Unit = {
  if (!stopped.get()) {
 val it = queues.iterator()
 while (it.hasNext()) {
it.next().post(event)
 }
  }
}
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

[Ngone51]

Github user Ngone51 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20039#discussion_r158573853
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf) {
 }
 
 this.sparkContext = sc
-queues.asScala.foreach(_.start(sc))
+queues.asScala.foreach { q =>
+  q.start(sc)
+  queuedEvents.foreach(q.post)
--- End diff --

Ok，**synchronized** can avoid this problem.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19982: [SPARK-22787] [TEST] [SQL] Add a TPC-H query suite

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/19982
  
@gatorsmile Any progress on this? 
https://github.com/apache/spark/pull/19982#discussion_r157119941
After I thought your comment, I came up with collecting metrics for each 
rule like;
https://github.com/apache/spark/compare/master...maropu:MetricSpike
This conflicts with your activity, or this is not acceptable? welcome any 
comment.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20011
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85320/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20011
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20011
  
**[Test build #85320 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85320/testReport)**
 for PR 20011 at commit 
[`be3f130`](https://github.com/apache/spark/commit/be3f1307f0edffdc7c9457ec960781cd28b07bf8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

[Ngone51]

Github user Ngone51 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20039#discussion_r158573360
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf) {
 }
 
 this.sparkContext = sc
-queues.asScala.foreach(_.start(sc))
+queues.asScala.foreach { q =>
+  q.start(sc)
+  queuedEvents.foreach(q.post)
--- End diff --

What if stop() called before all queuedEvents post to AsyncEventQueue?
```
/**
   * Stop the listener bus. It will wait until the queued events have been 
processed, but new
   * events will be dropped.
   */
```
**(the "queued  events" mentioned in description above is not equal to 
"queuedEvents" here.)**

As queuedEvents "post" before  listeners install, so, can they be treated  
as new events?



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20060: [SPARK-22889][SPARKR] Set overwrite=T when install Spark...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20060
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85328/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20060: [SPARK-22889][SPARKR] Set overwrite=T when install Spark...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20060
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20060: [SPARK-22889][SPARKR] Set overwrite=T when install Spark...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20060
  
**[Test build #85328 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85328/testReport)**
 for PR 20060 at commit 
[`6edbd71`](https://github.com/apache/spark/commit/6edbd710d0b73a7c4170dbb78eb42ace5a8c73a0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20059
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85321/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][K8s] Add documentation covering init conta...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20059
  
**[Test build #85321 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85321/testReport)**
 for PR 20059 at commit 
[`c0a659a`](https://github.com/apache/spark/commit/c0a659a3117674dfbd5c078badd653886c15cc8e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19929: [SPARK-22629][PYTHON] Add deterministic flag to pyspark ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19929
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85322/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19929: [SPARK-22629][PYTHON] Add deterministic flag to pyspark ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19929
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19929: [SPARK-22629][PYTHON] Add deterministic flag to pyspark ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19929
  
**[Test build #85322 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85322/testReport)**
 for PR 19929 at commit 
[`cc309b0`](https://github.com/apache/spark/commit/cc309b0ce2496365afd8c602c282e3d84aeed940).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/19977
  
I found the optimizer rule can't combine nested concat like;
```
scala>: psate
val df = sql("""
SELECT ((col1 || col2) || (col3 || col4)) col
FROM (
  SELECT
encode(string(id), 'utf-8') col1,
encode(string(id + 1), 'utf-8') col2,
string(id + 2) col3,
string(id + 3) col4
  FROM range(10)
)
""")

scala> df.explain(true)
== Parsed Logical Plan ==
'Project [concat(concat('col1, 'col2), concat('col3, 'col4)) AS col#4]+- 
'SubqueryAlias __auto_generated_subquery_name
   +- 'Project ['encode('string('id), utf-8) AS col1#0, 
'encode('string(('id + 1)), utf-8) AS col2#1, 'string(('id + 2)) AS col3#2, 
'string(('id + 3)) AS col4#3
]
  +- 'UnresolvedTableValuedFunction range, [10]

== Analyzed Logical Plan ==
col: string
Project [concat(cast(concat(col1#0, col2#1) as string), concat(col3#2, 
col4#3)) AS col#4]
+- SubqueryAlias __auto_generated_subquery_name
   +- Project [encode(cast(id#9L as string), utf-8) AS col1#0, 
encode(cast((id#9L + cast(1 as bigint)) as string), utf-8) AS col2#1, 
cast((id#9L + cast(2 as bigint)) as string) AS col3#2, cast((id#9L + cast(3 as 
bigint)) as string) AS col4#3]
  +- Range (0, 10, step=1, splits=None)

== Optimized Logical Plan ==
Project [concat(cast(concat(encode(cast(id#9L as string), utf-8), 
encode(cast((id#9L + 1) as string), utf-8)) as string), cast((id#9L + 2) as 
string), cast((id#9L + 3) as string)) AS col#4]
+- Range (0, 10, step=1, splits=None)

== Physical Plan ==
*Project [concat(cast(concat(encode(cast(id#9L as string), utf-8), 
encode(cast((id#9L + 1) as string), utf-8)) as string), cast((id#9L + 2) as 
string), cast((id#9L + 3) as string)) AS col#4]
+- *Range (0, 10, step=1, splits=4)
```
We need to support the optimization in this case, too?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85329 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85329/testReport)**
 for PR 19977 at commit 
[`3ec440d`](https://github.com/apache/spark/commit/3ec440de914d8f71c35f30cdccec815b199b5f17).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

[Ngone51]

Github user Ngone51 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20039#discussion_r158572288
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf) {
 }
 
 this.sparkContext = sc
-queues.asScala.foreach(_.start(sc))
+queues.asScala.foreach { q =>
+  q.start(sc)
+  queuedEvents.foreach(q.post)
--- End diff --

Yes，that‘s true, semantic only.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19977: [SPARK-22771][SQL] Concatenate binary inputs into...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/spark/pull/19977#discussion_r158572201
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2171,7 +2171,8 @@ object functions {
   def base64(e: Column): Column = withExpr { Base64(e.expr) }
 
   /**
-   * Concatenates multiple input string columns together into a single 
string column.
+   * Concatenates multiple input columns together into a single column.
+   * If all inputs are binary, concat returns an output as binary. 
Otherwise, it returns as string.
--- End diff --

ok


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20053: [SPARK-22873] [CORE] Init lastReportTimestamp wit...

[Ngone51]

Github user Ngone51 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20053#discussion_r158572161
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ---
@@ -112,6 +112,7 @@ private class AsyncEventQueue(val name: String, conf: 
SparkConf, metrics: LiveLi
   private[scheduler] def start(sc: SparkContext): Unit = {
 if (started.compareAndSet(false, true)) {
   this.sc = sc
+  lastReportTimestamp = System.currentTimeMillis()
--- End diff --

But there's no 'last time' for 'first time'.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20060: [SPARK-22889][SPARKR] Set overwrite=T when install Spark...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20060
  
**[Test build #85328 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85328/testReport)**
 for PR 20060 at commit 
[`6edbd71`](https://github.com/apache/spark/commit/6edbd710d0b73a7c4170dbb78eb42ace5a8c73a0).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20060: [SPARK-22889][SPARKR] Set overwrite=T when install Spark...

[shivaram]

Github user shivaram commented on the issue:

https://github.com/apache/spark/pull/20060
  
cc @felixcheung 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20060: [SPARK-22889][SPARKR] Set overwrite=T when instal...

[shivaram]

GitHub user shivaram opened a pull request:

https://github.com/apache/spark/pull/20060

[SPARK-22889][SPARKR] Set overwrite=T when install SparkR in tests

## What changes were proposed in this pull request?

Since all CRAN checks go through the same machine, if there is an older 
partial download or partial install of Spark left behind the tests fail. This 
PR overwrites the install files when running tests. This shouldn't affect 
Jenkins as `SPARK_HOME` is set when running Jenkins tests.

## How was this patch tested?

Test manually by running `R CMD check --as-cran`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shivaram/spark-1 sparkr-overwrite-cran

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20060.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20060


commit 6edbd710d0b73a7c4170dbb78eb42ace5a8c73a0
Author: Shivaram Venkataraman 
Date:   2017-12-23T00:27:56Z

Set overwrite=T when install SparkR in tests




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compres...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19218#discussion_r158571702
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala
 ---
@@ -42,8 +43,15 @@ private[parquet] class ParquetOptions(
* Acceptable values are defined in 
[[shortParquetCompressionCodecNames]].
*/
   val compressionCodecClassName: String = {
-val codecName = parameters.getOrElse("compression",
-  sqlConf.parquetCompressionCodec).toLowerCase(Locale.ROOT)
+// `compression`, `parquet.compression`(i.e., 
ParquetOutputFormat.COMPRESSION), and
+// `spark.sql.parquet.compression.codec`
+// are in order of precedence from highest to lowest.
+val parquetCompressionConf = 
parameters.get(ParquetOutputFormat.COMPRESSION)
+val codecName = parameters
+  .get("compression")
+  .orElse(parquetCompressionConf)
--- End diff --

Could we keep the old behavior? We could add it later? We do not want to 
mix multiple issues in the same PR? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compres...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19218#discussion_r158571657
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveOptions.scala 
---
@@ -102,4 +111,18 @@ object HiveOptions {
 "collectionDelim" -> "colelction.delim",
 "mapkeyDelim" -> "mapkey.delim",
 "lineDelim" -> "line.delim").map { case (k, v) => 
k.toLowerCase(Locale.ROOT) -> v }
+
+  def getHiveWriteCompression(tableInfo: TableDesc, sqlConf: SQLConf): 
Option[(String, String)] = {
+tableInfo.getOutputFileFormatClassName.toLowerCase match {
+  case formatName if formatName.endsWith("parquetoutputformat") =>
+val compressionCodec = new 
ParquetOptions(tableInfo.getProperties.asScala.toMap,
+  sqlConf).compressionCodecClassName
+Option((ParquetOutputFormat.COMPRESSION, compressionCodec))
+  case formatName if formatName.endsWith("orcoutputformat") =>
+val compressionCodec = new 
OrcOptions(tableInfo.getProperties.asScala.toMap,
+  sqlConf).compressionCodec
--- End diff --

Yeah. Just to make it consistent 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compres...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19218#discussion_r158571649
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -35,7 +39,7 @@ case class TestData(key: Int, value: String)
 case class ThreeCloumntable(key: Int, value: String, key1: String)
 
 class InsertSuite extends QueryTest with TestHiveSingleton with 
BeforeAndAfter
-with SQLTestUtils {
+with ParquetTest {
--- End diff --

Fine to me. Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19929: [SPARK-22629][PYTHON] Add deterministic flag to p...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19929#discussion_r158571445
  
--- Diff: python/pyspark/sql/functions.py ---
@@ -2075,9 +2075,10 @@ class PandasUDFType(object):
 def udf(f=None, returnType=StringType()):
--- End diff --

Do we need to just add a parameter for deterministic? Adding it to the end 
is OK to PySpark without breaking the existing app? cc @ueshin  


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19929: [SPARK-22629][PYTHON] Add deterministic flag to p...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19929#discussion_r158571371
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -58,6 +58,7 @@ class UDFRegistration private[sql] (functionRegistry: 
FunctionRegistry) extends
 | pythonIncludes: ${udf.func.pythonIncludes}
 | pythonExec: ${udf.func.pythonExec}
 | dataType: ${udf.dataType}
--- End diff --

Could you also print out `pythonEvalType`? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19929: [SPARK-22629][PYTHON] Add deterministic flag to p...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19929#discussion_r158571355
  
--- Diff: python/pyspark/sql/udf.py ---
@@ -157,5 +158,13 @@ def wrapper(*args):
 wrapper.func = self.func
 wrapper.returnType = self.returnType
 wrapper.evalType = self.evalType
+wrapper.asNondeterministic = self.asNondeterministic
 
 return wrapper
+
+def asNondeterministic(self):
+"""
+Updates UserDefinedFunction to nondeterministic.
--- End diff --

```
"""
Updates UserDefinedFunction to nondeterministic.

.. versionadded:: 2.3
"""
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19929: [SPARK-22629][PYTHON] Add deterministic flag to p...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19929#discussion_r158571320
  
--- Diff: python/pyspark/sql/tests.py ---
@@ -434,6 +434,16 @@ def test_udf_with_array_type(self):
 self.assertEqual(list(range(3)), l1)
 self.assertEqual(1, l2)
 
+def test_nondeterministic_udf(self):
+from pyspark.sql.functions import udf
+import random
+udf_random_col = udf(lambda: int(100 * random.random()), 
IntegerType()).asNondeterministic()
+df = 
self.spark.createDataFrame([Row(1)]).select(udf_random_col().alias('RAND'))
+random.seed(1234)
+udf_add_ten = udf(lambda rand: rand + 10, IntegerType())
+[row] = df.withColumn('RAND_PLUS_TEN', 
udf_add_ten('RAND')).collect()
+self.assertEqual(row[0] + 10, row[1])
--- End diff --

Compare the values, since you already set the seed?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19954
  
**[Test build #85327 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85327/testReport)**
 for PR 19954 at commit 
[`f82c568`](https://github.com/apache/spark/commit/f82c5682565bc0e1bc34ec428faedb53ee5ddecd).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20059: [SPARK-22648][Kubernetes] Update documentation to...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158570531
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,57 @@ by their appropriate remote URIs. Also, application 
dependencies can be pre-moun
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
 `SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
 
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like 
HDFS or HTTP servers, the driver and executor pods
+need a Kubernetes 
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
 for downloading
+the dependencies so the driver and executor containers can use them 
locally. This requires users to specify the container
+image for the init-container using the configuration property 
`spark.kubernetes.initContainer.image`. For example, users
+simply add the following option to the `spark-submit` command to specify 
the init-container image:
+
+```
+--conf spark.kubernetes.initContainer.image=
+```
+
+The init-container handles remote dependencies specified in `spark.jars` 
(or the `--jars` option of `spark-submit`) and
+`spark.files` (or the `--files` option of `spark-submit`). It also handles 
remotely hosted main application resources, e.g.,
+the main application jar. The following shows an example of using remote 
dependencies with the `spark-submit` command:
+
+```bash
+$ bin/spark-submit \
+--master k8s://https://: \
+--deploy-mode cluster \
+--name spark-pi \
+--class org.apache.spark.examples.SparkPi \
+--jars https://path/to/dependency1.jar,https://path/to/dependency2.jar
+--files hdfs://host:port/path/to/file1,hdfs://host:port/path/to/file2
+--conf spark.executor.instances=5 \
+--conf spark.kubernetes.driver.docker.image= \
+--conf spark.kubernetes.executor.docker.image= \
+--conf spark.kubernetes.initContainer.image=
+https://path/to/examples.jar
+```
+
+## Secret Management
+In some cases, a Spark application may need to use some credentials, e.g., 
for accessing data on a secured HDFS cluster
--- End diff --

I'd rewrite this.

"Kubernetes Secrets can be used to provide credentials for a Spark 
application to access secured services. To mount secrets into a driver 
container, ..."


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20059: [SPARK-22648][Kubernetes] Update documentation to...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158570653
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -528,51 +579,90 @@ specific to Spark on Kubernetes.
   
 
 
-   spark.kubernetes.driver.limit.cores
-   (none)
-   
- Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for the driver pod.
-   
- 
- 
-   spark.kubernetes.executor.limit.cores
-   (none)
-   
- Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for each executor pod launched for the Spark Application.
-   
- 
- 
-   spark.kubernetes.node.selector.[labelKey]
-   (none)
-   
- Adds to the node selector of the driver pod and executor pods, with 
key labelKey and the value as the
- configuration's value. For example, setting 
spark.kubernetes.node.selector.identifier to 
myIdentifier
- will result in the driver pod and executors having a node selector 
with key identifier and value
-  myIdentifier. Multiple node selector keys can be added 
by setting multiple configurations with this prefix.
-
-  
- 
-   
spark.kubernetes.driverEnv.[EnvironmentVariableName]
-   (none)
-   
- Add the environment variable specified by 
EnvironmentVariableName to
- the Driver process. The user can specify multiple of these to set 
multiple environment variables.
-   
- 
-  
-
spark.kubernetes.mountDependencies.jarsDownloadDir
-/var/spark-data/spark-jars
-
-  Location to download jars to in the driver and executors.
-  This directory must be empty and will be mounted as an empty 
directory volume on the driver and executor pods.
-
-  
-   
- 
spark.kubernetes.mountDependencies.filesDownloadDir
- /var/spark-data/spark-files
- 
-   Location to download jars to in the driver and executors.
-   This directory must be empty and will be mounted as an empty 
directory volume on the driver and executor pods.
- 
-   
+  spark.kubernetes.driver.limit.cores
+  (none)
+  
+Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for the driver pod.
+  
+
+
+  spark.kubernetes.executor.limit.cores
+  (none)
+  
+Specify the hard CPU 
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
 for each executor pod launched for the Spark Application.
+  
+
+
+  spark.kubernetes.node.selector.[labelKey]
+  (none)
+  
+Adds to the node selector of the driver pod and executor pods, with 
key labelKey and the value as the
+configuration's value. For example, setting 
spark.kubernetes.node.selector.identifier to 
myIdentifier
+will result in the driver pod and executors having a node selector 
with key identifier and value
+ myIdentifier. Multiple node selector keys can be added 
by setting multiple configurations with this prefix.
+  
+
+
+  
spark.kubernetes.driverEnv.[EnvironmentVariableName]
+  (none)
+  
+Add the environment variable specified by 
EnvironmentVariableName to
+the Driver process. The user can specify multiple of these to set 
multiple environment variables.
+  
+
+
+  spark.kubernetes.mountDependencies.jarsDownloadDir
+  /var/spark-data/spark-jars
+  
+Location to download jars to in the driver and executors.
+This directory must be empty and will be mounted as an empty directory 
volume on the driver and executor pods.
+  
+
+
+  spark.kubernetes.mountDependencies.filesDownloadDir
+  /var/spark-data/spark-files
+  
+Location to download jars to in the driver and executors.
+This directory must be empty and will be mounted as an empty directory 
volume on the driver and executor pods.
+  
+
+
+  spark.kubernetes.mountDependencies.timeout
+  5 minutes
+  
+   Timeout before aborting the attempt to download and unpack dependencies 
from remote locations into the driver and executor pods.
+  
+
+
+  
spark.kubernetes.mountDependencies.maxThreadPoolSize
+  5
+  
+   Maximum size of the thread pool for downloading remote dependencies 
into the driver and executor pods.
--- End diff --

I'd clarify this controls how many downloads happen simultaneously; could 
even change the name of the config 

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20011
  
**[Test build #85326 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85326/testReport)**
 for PR 20011 at commit 
[`931b2d2`](https://github.com/apache/spark/commit/931b2d262aa02880631ca4c693a84fa4c4d12318).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/20011#discussion_r158569095
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
 ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.File
+import java.nio.file.Files
+import java.nio.file.attribute.PosixFilePermissions
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.{HashMap, ListBuffer}
+
+import org.apache.commons.io.FileUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.status.KVUtils._
+import org.apache.spark.util.{Clock, Utils}
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * A class used to keep track of disk usage by the SHS, allowing 
application data to be deleted
+ * from disk when usage exceeds a configurable threshold.
+ *
+ * The goal of the class is not to guarantee that usage will never exceed 
the threshold; because of
+ * how application data is written, disk usage may temporarily go higher. 
But, eventually, it
+ * should fall back under the threshold.
+ *
+ * @param conf Spark configuration.
+ * @param path Path where to store application data.
+ * @param listing The listing store, used to persist usage data.
+ * @param clock Clock instance to use.
+ */
+private class HistoryServerDiskManager(
+conf: SparkConf,
+path: File,
+listing: KVStore,
+clock: Clock) extends Logging {
+
+  import config._
+
+  private val appStoreDir = new File(path, "apps")
+  if (!appStoreDir.isDirectory() && !appStoreDir.mkdir()) {
+throw new IllegalArgumentException(s"Failed to create app directory 
($appStoreDir).")
+  }
+
+  private val tmpStoreDir = new File(path, "temp")
+  if (!tmpStoreDir.isDirectory() && !tmpStoreDir.mkdir()) {
+throw new IllegalArgumentException(s"Failed to create temp directory 
($tmpStoreDir).")
+  }
+
+  private val maxUsage = conf.get(MAX_LOCAL_DISK_USAGE)
+  private val currentUsage = new AtomicLong(0L)
+  private val committedUsage = new AtomicLong(0L)
+  private val active = new HashMap[(String, Option[String]), Long]()
+
+  def initialize(): Unit = {
+updateUsage(sizeOf(appStoreDir), committed = true)
+
+// Clean up any temporary stores during start up. This assumes that 
they're leftover from other
+// instances and are not useful.
+tmpStoreDir.listFiles().foreach(FileUtils.deleteQuietly)
+
+// Go through the recorded store directories and remove any that may 
have been removed by
+// external code.
+val orphans = 
listing.view(classOf[ApplicationStoreInfo]).asScala.filter { info =>
+  !new File(info.path).exists()
+}.toSeq
+
+orphans.foreach { info =>
+  listing.delete(info.getClass(), info.path)
+}
+  }
+
+  /**
+   * Lease some space from the store. The leased space is calculated as a 
fraction of the given
+   * event log size; this is an approximation, and doesn't mean the 
application store cannot
+   * outgrow the lease.
+   *
+   * If there's not enough space for the lease, other applications might 
be evicted to make room.
+   * This method always returns a lease, meaning that it's possible for 
local disk usage to grow
+   * past the configured threshold if there aren't enough idle 
applications to evict.
+   *
+   * While the lease is active, the data is written to a temporary 
location, so `openStore()`
+   * will still return `None` for the application.
+   */
+  def lease(eventLogSize: Long, isCompressed: Boolean = false): Lease = {
+val needed = approximateSize(eventLogSize, isCompressed)
+makeRoom(needed)
+
+val perms = 

[GitHub] spark pull request #20059: [SPARK-22648][Kubernetes] Update documentation to...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158568498
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,23 @@ by their appropriate remote URIs. Also, application 
dependencies can be pre-moun
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
 `SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
 
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like 
HDFS or HTTP servers, the driver and executor pods need a Kubernetes 
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
 for downloading the dependencies so the driver and executor containers can use 
them locally. This requires users to specify the container image for the 
init-container using the configuration property 
`spark.kubernetes.initContainer.image`. For example, users simply add the 
following option to the `spark-submit` command to specify the init-container 
image:
--- End diff --

HDFS and HTTP sound good. We can cover GCS elsewhere. Line breaks were for 
ease of reviewing by others (being able to comment on individual lines) and for 
consistency with the rest of the docs. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

[squito]

Github user squito commented on a diff in the pull request:

https://github.com/apache/spark/pull/20011#discussion_r158568388
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
 ---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.File
+import java.nio.file.Files
+import java.nio.file.attribute.PosixFilePermissions
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.{HashMap, ListBuffer}
+
+import org.apache.commons.io.FileUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.status.KVUtils._
+import org.apache.spark.util.{Clock, Utils}
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * A class used to keep track of disk usage by the SHS, allowing 
application data to be deleted
+ * from disk when usage exceeds a configurable threshold.
+ *
+ * The goal of the class is not to guarantee that usage will never exceed 
the threshold; because of
+ * how application data is written, disk usage may temporarily go higher. 
But, eventually, it
+ * should fall back under the threshold.
+ *
+ * @param conf Spark configuration.
+ * @param path Path where to store application data.
+ * @param listing The listing store, used to persist usage data.
+ * @param clock Clock instance to use.
+ */
+private class HistoryServerDiskManager(
+conf: SparkConf,
+path: File,
+listing: KVStore,
+clock: Clock) extends Logging {
+
+  import config._
+
+  private val appStoreDir = new File(path, "apps")
+  if (!appStoreDir.isDirectory() && !appStoreDir.mkdir()) {
+throw new IllegalArgumentException(s"Failed to create app directory 
($appStoreDir).")
+  }
+
+  private val tmpStoreDir = new File(path, "temp")
+  if (!tmpStoreDir.isDirectory() && !tmpStoreDir.mkdir()) {
+throw new IllegalArgumentException(s"Failed to create temp directory 
($tmpStoreDir).")
+  }
+
+  private val maxUsage = conf.get(MAX_LOCAL_DISK_USAGE)
+  private val currentUsage = new AtomicLong(0L)
+  private val committedUsage = new AtomicLong(0L)
+  private val active = new HashMap[(String, Option[String]), Long]()
+
+  def initialize(): Unit = {
+updateUsage(sizeOf(appStoreDir), committed = true)
+
+// Clean up any temporary stores during start up. This assumes that 
they're leftover from other
+// instances and are not useful.
+tmpStoreDir.listFiles().foreach(FileUtils.deleteQuietly)
+
+// Go through the recorded store directories and remove any that may 
have been removed by
+// external code.
+val orphans = 
listing.view(classOf[ApplicationStoreInfo]).asScala.filter { info =>
+  !new File(info.path).exists()
+}.toSeq
+
+orphans.foreach { info =>
+  listing.delete(info.getClass(), info.path)
+}
+  }
+
+  /**
+   * Lease some space from the store. The leased space is calculated as a 
fraction of the given
+   * event log size; this is an approximation, and doesn't mean the 
application store cannot
+   * outgrow the lease.
+   *
+   * If there's not enough space for the lease, other applications might 
be evicted to make room.
+   * This method always returns a lease, meaning that it's possible for 
local disk usage to grow
+   * past the configured threshold if there aren't enough idle 
applications to evict.
+   *
+   * While the lease is active, the data is written to a temporary 
location, so `openStore()`
+   * will still return `None` for the application.
+   */
+  def lease(eventLogSize: Long, isCompressed: Boolean = false): Lease = {
+val needed = approximateSize(eventLogSize, isCompressed)
+makeRoom(needed)
+
+val perms = 

[GitHub] spark issue #20059: [SPARK-22648][Kubernetes] Update documentation to cover ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20059
  
**[Test build #85325 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85325/testReport)**
 for PR 20059 at commit 
[`fbb2112`](https://github.com/apache/spark/commit/fbb21121447fe131358042ce05454f88d6fb).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][Kubernetes] Update documentation to cover ...

[liyinan926]

Github user liyinan926 commented on the issue:

https://github.com/apache/spark/pull/20059
  
Updated in 
https://github.com/apache/spark/pull/20059/commits/fbb21121447fe131358042ce05454f88d6fb.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20039
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85317/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20039
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20059: [SPARK-22648][Kubernetes] Update documentation to...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158568089
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,23 @@ by their appropriate remote URIs. Also, application 
dependencies can be pre-moun
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
 `SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
 
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like 
HDFS or HTTP servers, the driver and executor pods need a Kubernetes 
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
 for downloading the dependencies so the driver and executor containers can use 
them locally. This requires users to specify the container image for the 
init-container using the configuration property 
`spark.kubernetes.initContainer.image`. For example, users simply add the 
following option to the `spark-submit` command to specify the init-container 
image:
--- End diff --

Regarding examples, I can add one spark-submit example showing how to use 
remote jars/files on http/https and hdfs. But gcs requires the connector in the 
init-container, which is non-trivial. I'm not sure about s3. I think we should 
avoid doing so.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20039
  
**[Test build #85317 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85317/testReport)**
 for PR 20039 at commit 
[`2602fa6`](https://github.com/apache/spark/commit/2602fa68424e984f2cd49f79fb54bcf9676ba5fb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18754: [SPARK-21552][SQL] Add DecimalType support to Arr...

[BryanCutler]

Github user BryanCutler commented on a diff in the pull request:

https://github.com/apache/spark/pull/18754#discussion_r158566190
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala 
---
@@ -214,6 +216,22 @@ private[arrow] class DoubleWriter(val valueVector: 
Float8Vector) extends ArrowFi
   }
 }
 
+private[arrow] class DecimalWriter(
+val valueVector: DecimalVector,
+precision: Int,
+scale: Int) extends ArrowFieldWriter {
+
+  override def setNull(): Unit = {
+valueVector.setNull(count)
+  }
+
+  override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {
+val decimal = input.getDecimal(ordinal, precision, scale)
+decimal.changePrecision(precision, scale)
--- End diff --

Is it necessary to call `changePrecision` even though `getDecimal` already 
takes the precision/scale as input - is it not guaranteed to return a decimal 
with that scale?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18754: [SPARK-21552][SQL] Add DecimalType support to Arr...

[BryanCutler]

Github user BryanCutler commented on a diff in the pull request:

https://github.com/apache/spark/pull/18754#discussion_r158567491
  
--- Diff: python/pyspark/sql/types.py ---
@@ -1617,7 +1617,7 @@ def to_arrow_type(dt):
 elif type(dt) == DoubleType:
 arrow_type = pa.float64()
 elif type(dt) == DecimalType:
-arrow_type = pa.decimal(dt.precision, dt.scale)
+arrow_type = pa.decimal128(dt.precision, dt.scale)
--- End diff --

yes, that's the right way - it is now fixed at 128 bits internally.  I 
believe the Arrow Java limit is the same as Spark 38/38, not sure if pyarrow is 
the same but I think so.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20045: [Spark-22360][SQL] Add unit tests for Window Specificati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20045
  
**[Test build #4023 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4023/testReport)**
 for PR 20045 at commit 
[`797c907`](https://github.com/apache/spark/commit/797c90739a4fc57f487d4d73c28ba59bfd598942).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20059: [SPARK-22648][Kubernetes] Update documentation to...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158567222
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,23 @@ by their appropriate remote URIs. Also, application 
dependencies can be pre-moun
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
 `SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
 
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like 
HDFS or HTTP servers, the driver and executor pods need a Kubernetes 
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
 for downloading the dependencies so the driver and executor containers can use 
them locally. This requires users to specify the container image for the 
init-container using the configuration property 
`spark.kubernetes.initContainer.image`. For example, users simply add the 
following option to the `spark-submit` command to specify the init-container 
image:
--- End diff --

Do we need to break them into lines? I thought this should be automatically 
wrapped when being viewed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19954
  
**[Test build #85324 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85324/testReport)**
 for PR 19954 at commit 
[`785b90e`](https://github.com/apache/spark/commit/785b90e52580e9175896b22b00b23f30fbe020ef).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20059: [SPARK-22648][Kubernetes] Update documentation to...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158566909
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,23 @@ by their appropriate remote URIs. Also, application 
dependencies can be pre-moun
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
 `SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
 
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like 
HDFS or HTTP servers, the driver and executor pods need a Kubernetes 
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
 for downloading the dependencies so the driver and executor containers can use 
them locally. This requires users to specify the container image for the 
init-container using the configuration property 
`spark.kubernetes.initContainer.image`. For example, users simply add the 
following option to the `spark-submit` command to specify the init-container 
image:
--- End diff --

Maybe we should include 2-3 examples of remote file usage - ideally, 
showing that one can use http, hdfs, gcs, s3 in dependencies.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20059: [SPARK-22648][Kubernetes] Update documentation to...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20059#discussion_r158566870
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,23 @@ by their appropriate remote URIs. Also, application 
dependencies can be pre-moun
 Those dependencies can be added to the classpath by referencing them with 
`local://` URIs and/or setting the
 `SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
 
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like 
HDFS or HTTP servers, the driver and executor pods need a Kubernetes 
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
 for downloading the dependencies so the driver and executor containers can use 
them locally. This requires users to specify the container image for the 
init-container using the configuration property 
`spark.kubernetes.initContainer.image`. For example, users simply add the 
following option to the `spark-submit` command to specify the init-container 
image:
--- End diff --

This and below text should be broken up into multiple lines.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20059: [SPARK-22648][Kubernetes] Update documentation to cover ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20059
  
**[Test build #85323 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85323/testReport)**
 for PR 20059 at commit 
[`a6a6660`](https://github.com/apache/spark/commit/a6a666060787cadd832b5bd32281940a6c81a9a9).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org