[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15659
  
**[Test build #68557 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68557/consoleFull)**
 for PR 15659 at commit 
[`3345eb9`](https://github.com/apache/spark/commit/3345eb9bd3f0a165ae1d20d76e5d36e88e5512cd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13837: [SPARK-16126] [SQL] Better Error Message When using Data...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13837
  
**[Test build #68556 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68556/consoleFull)**
 for PR 13837 at commit 
[`635046a`](https://github.com/apache/spark/commit/635046a10cc059a6ae8756fb7bc7167f5621255c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13837: [SPARK-16126] [SQL] Better Error Message When usi...

[gatorsmile]

GitHub user gatorsmile reopened a pull request:

https://github.com/apache/spark/pull/13837

[SPARK-16126] [SQL] Better Error Message When using DataFrameReader without 
`path`

 What changes were proposed in this pull request?

When users do not specify the path in `DataFrameReader` APIs, it can get a 
confusing error message. For example, 

``` Scala
spark.read.json()
```

Error message:

```
Unable to infer schema for JSON at . It must be specified manually;
```

After the fix, the error message will be like: 

```
'path' is not specified
```

Another major goal of this PR is to add test cases for the latest changes 
in https://github.com/apache/spark/pull/13727. 
- orc read APIs
- illegal format name
- save API - empty path or illegal path
- load API - empty path
- illegal compression
- fixed a test case in the existing test case `prevent all column 
partitioning`
 How was this patch tested?

Test cases are added.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark dfWriterAudit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13837.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13837


commit 8d021e47e9a4e95ade99d617c77ef1e17245a796
Author: gatorsmile 
Date:   2016-06-17T18:24:42Z

test cases

commit 5e4a3c666dfb767215130df1a778e5f97d438c54
Author: gatorsmile 
Date:   2016-06-17T19:58:56Z

add test cases.

commit 26437151ff0db4c0010510de047f81b1808890f4
Author: gatorsmile 
Date:   2016-06-17T23:48:23Z

fix and test cases

commit cfc0188a0baa45aef1bae6604dd10450eaafd561
Author: gatorsmile 
Date:   2016-06-21T01:59:02Z

Merge remote-tracking branch 'upstream/master' into dfWriterAudit

commit 3007fe66d03a6a40dc530c13d44c27030118a8a4
Author: gatorsmile 
Date:   2016-06-21T13:27:16Z

more test case

commit a1ae7249322c17ea09be4e968535dc115b2acb64
Author: gatorsmile 
Date:   2016-06-22T06:12:56Z

fix test case

commit 635046a10cc059a6ae8756fb7bc7167f5621255c
Author: gatorsmile 
Date:   2016-06-22T16:04:51Z

fix test case




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/15704#discussion_r87688739
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -418,27 +419,66 @@ case class AlterTableRenamePartitionCommand(
  */
 case class AlterTableDropPartitionCommand(
 tableName: TableIdentifier,
-specs: Seq[TablePartitionSpec],
+specs: Seq[Expression],
 ifExists: Boolean,
 purge: Boolean)
-  extends RunnableCommand {
+  extends RunnableCommand with PredicateHelper {
+
+  private def isRangeComparison(expr: Expression): Boolean = {
+expr.find(e => e.isInstanceOf[BinaryComparison] && 
!e.isInstanceOf[EqualTo]).isDefined
+  }
 
   override def run(sparkSession: SparkSession): Seq[Row] = {
 val catalog = sparkSession.sessionState.catalog
 val table = catalog.getTableMetadata(tableName)
+val resolver = sparkSession.sessionState.conf.resolver
 DDLUtils.verifyAlterTableType(catalog, table, isView = false)
 DDLUtils.verifyPartitionProviderIsHive(sparkSession, table, "ALTER 
TABLE DROP PARTITION")
 
-val normalizedSpecs = specs.map { spec =>
-  PartitioningUtils.normalizePartitionSpec(
-spec,
-table.partitionColumnNames,
-table.identifier.quotedString,
-sparkSession.sessionState.conf.resolver)
+specs.foreach { expr =>
+  expr.references.foreach { attr =>
+if (!table.partitionColumnNames.exists(resolver(_, attr.name))) {
+  throw new AnalysisException(s"${attr.name} is not a valid 
partition column " +
+s"in table ${table.identifier.quotedString}.")
+}
+  }
 }
 
-catalog.dropPartitions(
-  table.identifier, normalizedSpecs, ignoreIfNotExists = ifExists, 
purge = purge)
+if (specs.exists(isRangeComparison)) {
+  if (!ifExists) {
+// Prevent query execution if one of partition specs is invalid.
+specs.foreach { spec =>
+  val partitions = 
catalog.listPartitionsByFilter(table.identifier, Seq(spec))
--- End diff --

Yep, correct! Thank you so much, @viirya . Then, I'll update the PR like 
that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/15659
  
So the simplest thing sounds like updating the error message, for now I'll 
avoid changing the maven build since that might have some unintended collateral 
effects and this change is currently primarily additive.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/15563#discussion_r87688574
  
--- Diff: 
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -207,6 +207,10 @@ package object config {
 .booleanConf
 .createWithDefault(false)
 
+  private[spark] val APP_CALLER_CONTEXT = 
ConfigBuilder("spark.log.callerContext")
--- End diff --

Thanks for the comment. Would it be a problem if we use 
spark.hadoop.log.callerContext? I know it gets passed into Configuration, but 
why would that be a problem? Is it overriding some common configuration in 
Hadoop?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15717: [SPARK-17910][SQL] Allow users to update the comment of ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15717
  
Any update on this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15854
  
**[Test build #68555 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68555/consoleFull)**
 for PR 15854 at commit 
[`9ca806e`](https://github.com/apache/spark/commit/9ca806e2cac05610d43ffd6019c23b54ead3b40b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/15704#discussion_r87688495
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -418,27 +419,66 @@ case class AlterTableRenamePartitionCommand(
  */
 case class AlterTableDropPartitionCommand(
 tableName: TableIdentifier,
-specs: Seq[TablePartitionSpec],
+specs: Seq[Expression],
 ifExists: Boolean,
 purge: Boolean)
-  extends RunnableCommand {
+  extends RunnableCommand with PredicateHelper {
+
+  private def isRangeComparison(expr: Expression): Boolean = {
+expr.find(e => e.isInstanceOf[BinaryComparison] && 
!e.isInstanceOf[EqualTo]).isDefined
+  }
 
   override def run(sparkSession: SparkSession): Seq[Row] = {
 val catalog = sparkSession.sessionState.catalog
 val table = catalog.getTableMetadata(tableName)
+val resolver = sparkSession.sessionState.conf.resolver
 DDLUtils.verifyAlterTableType(catalog, table, isView = false)
 DDLUtils.verifyPartitionProviderIsHive(sparkSession, table, "ALTER 
TABLE DROP PARTITION")
 
-val normalizedSpecs = specs.map { spec =>
-  PartitioningUtils.normalizePartitionSpec(
-spec,
-table.partitionColumnNames,
-table.identifier.quotedString,
-sparkSession.sessionState.conf.resolver)
+specs.foreach { expr =>
+  expr.references.foreach { attr =>
+if (!table.partitionColumnNames.exists(resolver(_, attr.name))) {
+  throw new AnalysisException(s"${attr.name} is not a valid 
partition column " +
+s"in table ${table.identifier.quotedString}.")
+}
+  }
 }
 
-catalog.dropPartitions(
-  table.identifier, normalizedSpecs, ignoreIfNotExists = ifExists, 
purge = purge)
+if (specs.exists(isRangeComparison)) {
+  if (!ifExists) {
+// Prevent query execution if one of partition specs is invalid.
+specs.foreach { spec =>
+  val partitions = 
catalog.listPartitionsByFilter(table.identifier, Seq(spec))
--- End diff --

I think it is good. Actually the partitions dropped in the end should be 
the same. The difference is only if an exception is thrown or not, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE us...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/15854#discussion_r87688482
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -33,7 +33,9 @@ import org.apache.spark.sql.types.MetadataBuilder
  * ViewType is used to specify the expected view type when we want to 
create or replace a view in
  * [[CreateViewCommand]].
  */
-sealed trait ViewType
+sealed trait ViewType {
+  override def toString: String = getClass.getSimpleName.filter(_ != '$')
--- End diff --

Yeah!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15659
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15659
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68554/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15659
  
**[Test build #68554 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68554/consoleFull)**
 for PR 15659 at commit 
[`2904998`](https://github.com/apache/spark/commit/290499848cf9d65fa35e8488f11531091a014081).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...

[weiqingy]

Github user weiqingy commented on a diff in the pull request:

https://github.com/apache/spark/pull/15563#discussion_r87688398
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2578,26 +2579,38 @@ private[util] object CallerContext extends Logging {
  * @param taskAttemptNumber task attempt id
  */
 private[spark] class CallerContext(
-   from: String,
-   appId: Option[String] = None,
-   appAttemptId: Option[String] = None,
-   jobId: Option[Int] = None,
-   stageId: Option[Int] = None,
-   stageAttemptId: Option[Int] = None,
-   taskId: Option[Long] = None,
-   taskAttemptNumber: Option[Int] = None) extends Logging {
-
-   val appIdStr = if (appId.isDefined) s"_${appId.get}" else ""
-   val appAttemptIdStr = if (appAttemptId.isDefined) 
s"_${appAttemptId.get}" else ""
-   val jobIdStr = if (jobId.isDefined) s"_JId_${jobId.get}" else ""
-   val stageIdStr = if (stageId.isDefined) s"_SId_${stageId.get}" else ""
-   val stageAttemptIdStr = if (stageAttemptId.isDefined) 
s"_${stageAttemptId.get}" else ""
-   val taskIdStr = if (taskId.isDefined) s"_TId_${taskId.get}" else ""
-   val taskAttemptNumberStr =
- if (taskAttemptNumber.isDefined) s"_${taskAttemptNumber.get}" else ""
-
-   val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
- jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + 
taskAttemptNumberStr
+  from: String,
--- End diff --

What do you mean "off"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...

[weiqingy]

Github user weiqingy commented on a diff in the pull request:

https://github.com/apache/spark/pull/15563#discussion_r87688391
  
--- Diff: 
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -207,6 +207,10 @@ package object config {
 .booleanConf
 .createWithDefault(false)
 
+  private[spark] val APP_CALLER_CONTEXT = 
ConfigBuilder("spark.log.callerContext")
--- End diff --

This is not just for Yarn, if running spark apps in standalonde mode with 
master and workers and reading/writing from/to hdfs, the caller context would 
still work on the hdfs side. (PS. we also can not use spark.hadoop prefix that 
will be treated as Hadoop configuration and set to `Configuration`.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15659
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15659
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68553/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15659
  
**[Test build #68553 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68553/consoleFull)**
 for PR 15659 at commit 
[`587c0eb`](https://github.com/apache/spark/commit/587c0ebc71ff587e48ec8e71728069d155694dd2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15860
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15860
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68552/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15860
  
**[Test build #68552 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68552/consoleFull)**
 for PR 15860 at commit 
[`f47101a`](https://github.com/apache/spark/commit/f47101aedbf6c5d9dbf2ca7f1b6456b1e867a438).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[minrk]

Github user minrk commented on the issue:

https://github.com/apache/spark/pull/15659
  
> Any ideas on how we could follow up and support that install pattern?

To go all the way to the two commands I expected to work based on Python 
tradition:

```bash
./build/mvn -DskipTests clean package
cd python && pip install .
```

I believe you would need to copy (or hardlink or symlink-to-absolute-path) 
the jars into the Python directory during `build\mvn`, since `pip` copies the 
whole `python` directory to a temp location before running your setup.py.

One step short of this would be to allow the `stage-jars` to be an explicit 
step, rather than a temporary staging in setup.py that cleans up after itself. 
Then you would have one small extra step and the full command would be:

```bash
./build/mvn
cd python
python setup.py stage_jars # ./stage-jars.sh, whatever you prefer
pip install .
```

and the error message when jars are missing could point to the extra step.

Yet another option, and the least work from what you have now, could be to 
put the full sequence in your missing-jars error message:

> If you are installing pyspark from the spark source, you must build spark 
and run sdist first:
> ```
> ./build/mvn -DskipTests clean package
> cd python
> python setup.py sdist
> pip install dist/*.tar.gz
> ```

I think most people wouldn't discover that without very explicit help, 
because they would need to know about both the temporary symlinks and pip's 
tempdir to work it out.

Not being able to do `pip install .` is a bit odd for a Python package, as 
that is the standard command for installing any Python package from source 
(it's not `python setup.py install` anymore). But if you provide something 
copy/pasteable when people do try it, you are in pretty good shape.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE us...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/15854#discussion_r87687899
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -33,7 +33,9 @@ import org.apache.spark.sql.types.MetadataBuilder
  * ViewType is used to specify the expected view type when we want to 
create or replace a view in
  * [[CreateViewCommand]].
  */
-sealed trait ViewType
+sealed trait ViewType {
+  override def toString: String = getClass.getSimpleName.filter(_ != '$')
--- End diff --

stripSuffix?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/15563#discussion_r87687846
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2578,26 +2579,38 @@ private[util] object CallerContext extends Logging {
  * @param taskAttemptNumber task attempt id
  */
 private[spark] class CallerContext(
-   from: String,
-   appId: Option[String] = None,
-   appAttemptId: Option[String] = None,
-   jobId: Option[Int] = None,
-   stageId: Option[Int] = None,
-   stageAttemptId: Option[Int] = None,
-   taskId: Option[Long] = None,
-   taskAttemptNumber: Option[Int] = None) extends Logging {
-
-   val appIdStr = if (appId.isDefined) s"_${appId.get}" else ""
-   val appAttemptIdStr = if (appAttemptId.isDefined) 
s"_${appAttemptId.get}" else ""
-   val jobIdStr = if (jobId.isDefined) s"_JId_${jobId.get}" else ""
-   val stageIdStr = if (stageId.isDefined) s"_SId_${stageId.get}" else ""
-   val stageAttemptIdStr = if (stageAttemptId.isDefined) 
s"_${stageAttemptId.get}" else ""
-   val taskIdStr = if (taskId.isDefined) s"_TId_${taskId.get}" else ""
-   val taskAttemptNumberStr =
- if (taskAttemptNumber.isDefined) s"_${taskAttemptNumber.get}" else ""
-
-   val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
- jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + 
taskAttemptNumberStr
+  from: String,
--- End diff --

this is off?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/15659
  
Thanks @JoshRosen for taking time to review this. The factoured out fix is 
at https://github.com/apache/spark/pull/15860 :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15763
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68551/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15763
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15763
  
**[Test build #68551 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68551/consoleFull)**
 for PR 15763 at commit 
[`7722503`](https://github.com/apache/spark/commit/7722503e68981f340e8a44aa91d5738b6d6a6b2a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `public class AesCipher `
  * `public class AesConfigMessage implements Encodable `
  * `public class ByteArrayReadableChannel implements ReadableByteChannel `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15704
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15704
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68550/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15704
  
**[Test build #68550 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68550/consoleFull)**
 for PR 15704 at commit 
[`f3f0ad5`](https://github.com/apache/spark/commit/f3f0ad546a880fa36b8f62ee67eeedd6a79bda1b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15659
  
**[Test build #68554 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68554/consoleFull)**
 for PR 15659 at commit 
[`2904998`](https://github.com/apache/spark/commit/290499848cf9d65fa35e8488f11531091a014081).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/15659
  
@minrk thanks :) Any ideas on how we could follow up and support that 
install pattern?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15659
  
**[Test build #68553 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68553/consoleFull)**
 for PR 15659 at commit 
[`587c0eb`](https://github.com/apache/spark/commit/587c0ebc71ff587e48ec8e71728069d155694dd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/15659#discussion_r87686589
  
--- Diff: pom.xml ---
@@ -26,6 +26,7 @@
   
   org.apache.spark
   spark-parent_2.11
+  
--- End diff --

@JoshRosen so we already update this implicitly using release-tag.sh - this 
is just the version for dev builds.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15860
  
**[Test build #68552 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68552/consoleFull)**
 for PR 15860 at commit 
[`f47101a`](https://github.com/apache/spark/commit/f47101aedbf6c5d9dbf2ca7f1b6456b1e867a438).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15840: [SPARK-18398][SQL] Fix nullabilities of MapObjects and o...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/15840
  
I agree that this case is fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/15659
  
@joshrosen Is `shellcheck` intended to be part of our standard linting?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/15659#discussion_r87686249
  
--- Diff: dev/create-release/release-build.sh ---
@@ -187,10 +208,10 @@ if [[ "$1" == "package" ]]; then
   # We increment the Zinc port each time to avoid OOM's and other 
craziness if multiple builds
   # share the same Zinc server.
   FLAGS="-Psparkr -Phive -Phive-thriftserver -Pyarn -Pmesos"
-  make_binary_release "hadoop2.3" "-Phadoop2.3 $FLAGS" "3033" &
-  make_binary_release "hadoop2.4" "-Phadoop2.4 $FLAGS" "3034" &
-  make_binary_release "hadoop2.6" "-Phadoop2.6 $FLAGS" "3035" &
-  make_binary_release "hadoop2.7" "-Phadoop2.7 $FLAGS" "3036" &
+  make_binary_release "hadoop2.3" "-Phadoop-2.3 $FLAGS" "3033" &
+  make_binary_release "hadoop2.4" "-Phadoop-2.4 $FLAGS" "3034" &
+  make_binary_release "hadoop2.6" "-Phadoop-2.6 $FLAGS" "3035" &
+  make_binary_release "hadoop2.7" "-Phadoop-2.7 $FLAGS" "3036" &
--- End diff --

Done - https://github.com/apache/spark/pull/15860


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15860: [SPARK-18418] Fix flags for make_binary_release f...

[holdenk]

GitHub user holdenk opened a pull request:

https://github.com/apache/spark/pull/15860

[SPARK-18418] Fix flags for make_binary_release for hadoop profile

## What changes were proposed in this pull request?

Fix the flags used to specify the hadoop version

## How was this patch tested?

Manually tested as part of https://github.com/apache/spark/pull/15659 by 
having the build succeed.

cc @joshrosen

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/spark minor-fix-release-build-script

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15860.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15860


commit f47101aedbf6c5d9dbf2ca7f1b6456b1e867a438
Author: Holden Karau 
Date:   2016-11-12T04:09:47Z

Fix flags for make_binary_release for hadoop profile - split from 
https://github.com/apache/spark/pull/15659




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68548/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15763
  
**[Test build #68551 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68551/consoleFull)**
 for PR 15763 at commit 
[`7722503`](https://github.com/apache/spark/commit/7722503e68981f340e8a44aa91d5738b6d6a6b2a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68548 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68548/consoleFull)**
 for PR 15859 at commit 
[`7ae7286`](https://github.com/apache/spark/commit/7ae72863ce911e59b5a98841e80b37e51e3167ea).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68547/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68549/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68547 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68547/consoleFull)**
 for PR 15859 at commit 
[`0e3ef54`](https://github.com/apache/spark/commit/0e3ef545687bc9e57e001c9d55519d150e875dcb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68549 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68549/consoleFull)**
 for PR 15859 at commit 
[`b60485e`](https://github.com/apache/spark/commit/b60485e7b3b76e33f46c3c017733f2eaed57f922).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15854
  
@hvanhovell The PR description is updated. How about the latest change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15854
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68544/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15838: [SPARK-18396][HISTORYSERVER]"Duration" column makes sear...

[WangTaoTheTonic]

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/spark/pull/15838
  
Is it good to go? @srowen @vanzin 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15854
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15854
  
**[Test build #68544 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68544/consoleFull)**
 for PR 15854 at commit 
[`d6a3b3f`](https://github.com/apache/spark/commit/d6a3b3f6ebad8256efe6687d6adcaf3971b90ee4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15840: [SPARK-18398][SQL] Fix nullabilities of MapObjects and o...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/15840
  
@kiszk I have not checked all the case yet but I think the case that we 
need to generate else-clause doesn't match the case we discuss here.
Of course we can add the method you suggested when we find there is the 
case and let me know if you find it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68546/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15859
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68546 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68546/consoleFull)**
 for PR 15859 at commit 
[`fd564e6`](https://github.com/apache/spark/commit/fd564e62d9f8563b89bd2247d25944bda355b14c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/13909#discussion_r87685165
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala
 ---
@@ -43,11 +43,38 @@ trait ExpressionEvalHelper extends 
GeneratorDrivenPropertyChecks {
 
   protected def checkEvaluation(
   expression: => Expression, expected: Any, inputRow: InternalRow = 
EmptyRow): Unit = {
-val catalystValue = CatalystTypeConverters.convertToCatalyst(expected)
+// No codegen version expects GenericArrayData
+val catalystValue = expected match {
--- End diff --

This is still looks weird to me.

Actually you need this because you replace `Seq` with `Array` in 
`test("CreateArray")` at `ComplexTypeSuite.scala`.

Can we not change `Seq` to `Array` in `ComplexTypeSuite`?  We actually can 
convert `Seq` to `Array` for unsafe version only in `convertToCatalystUnsafe` 
if we move `convertToCatalystUnsafe` to `ExpressionEvalHelper.scala`.

No other place uses this `convertToCatalystUnsafe`, I think.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15563: [SPARK-16759][CORE] Add a configuration property to pass...

[weiqingy]

Github user weiqingy commented on the issue:

https://github.com/apache/spark/pull/15563
  
Thanks a lot for the review, @tgravescs @mridulm 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15704
  
**[Test build #68550 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68550/consoleFull)**
 for PR 15704 at commit 
[`f3f0ad5`](https://github.com/apache/spark/commit/f3f0ad546a880fa36b8f62ee67eeedd6a79bda1b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/15704#discussion_r87684919
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -225,6 +226,111 @@ class HiveDDLSuite
 }
   }
 
+  test("SPARK-17732: Drop partitions by filter") {
+withTable("sales") {
+  sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, 
quarter STRING)")
+
+  for (country <- Seq("US", "CA", "KR")) {
+for (quarter <- 1 to 4) {
+  sql(s"ALTER TABLE sales ADD PARTITION (country = '$country', 
quarter = '$quarter')")
+}
+  }
+
+  sql("ALTER TABLE sales DROP PARTITION (country < 'KR', quarter > 
'2')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=CA/quarter=1") ::
+Row("country=CA/quarter=2") ::
+Row("country=KR/quarter=1") ::
+Row("country=KR/quarter=2") ::
+Row("country=KR/quarter=3") ::
+Row("country=KR/quarter=4") ::
+Row("country=US/quarter=1") ::
+Row("country=US/quarter=2") ::
+Row("country=US/quarter=3") ::
+Row("country=US/quarter=4") :: Nil)
+
+  sql("ALTER TABLE sales DROP PARTITION (country < 'KR'), PARTITION 
(quarter <= '1')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=KR/quarter=2") ::
+Row("country=KR/quarter=3") ::
+Row("country=KR/quarter=4") ::
+Row("country=US/quarter=2") ::
+Row("country=US/quarter=3") ::
+Row("country=US/quarter=4") :: Nil)
+
+  sql("ALTER TABLE sales DROP PARTITION (country='KR', quarter='4')")
+  sql("ALTER TABLE sales DROP PARTITION (country='US', quarter='3')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=KR/quarter=2") ::
+Row("country=KR/quarter=3") ::
+Row("country=US/quarter=2") ::
+Row("country=US/quarter=4") :: Nil)
+
+  sql("ALTER TABLE sales DROP PARTITION (quarter <= 2), PARTITION 
(quarter >= '4')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=KR/quarter=3") :: Nil)
+
+  val m = intercept[AnalysisException] {
+sql("ALTER TABLE sales DROP PARTITION (quarter <= 4), PARTITION 
(quarter <= '3')")
+  }.getMessage
+  // `PARTITION (quarter <= '2')` should raises exceptions because 
`PARTITION (quarter <= 4)`
--- End diff --

Thanks. I fixed that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15563


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15563: [SPARK-16759][CORE] Add a configuration property to pass...

[mridulm]

Github user mridulm commented on the issue:

https://github.com/apache/spark/pull/15563
  
Merging into master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/15856
  
Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder shoul...

[dongjoon-hyun]

Github user dongjoon-hyun closed the pull request at:

https://github.com/apache/spark/pull/15856


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15857: [SPARK-18300][SQL] Do not apply foldable propagation wit...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15857
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15857: [SPARK-18300][SQL] Do not apply foldable propagation wit...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15857
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68543/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15857: [SPARK-18300][SQL] Do not apply foldable propagation wit...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15857
  
**[Test build #68543 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68543/consoleFull)**
 for PR 15857 at commit 
[`d98f8f9`](https://github.com/apache/spark/commit/d98f8f930330b368fbeade07d435197b0cdd2228).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/13909#discussion_r87684528
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
 ---
@@ -421,6 +421,17 @@ object CatalystTypeConverters {
 case other => other
   }
 
+  def convertToCatalystUnsafe(a: Any): Any = a match {
--- End diff --

Actually do we want to have this in `CatalystTypeConverters`?

I think it should be in `ExpressionEvalHelper.scala`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15563: [SPARK-16759][CORE] Add a configuration property to pass...

[mridulm]

Github user mridulm commented on the issue:

https://github.com/apache/spark/pull/15563
  
Looks good to me


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/15704#discussion_r87683697
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 
---
@@ -225,6 +226,111 @@ class HiveDDLSuite
 }
   }
 
+  test("SPARK-17732: Drop partitions by filter") {
+withTable("sales") {
+  sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, 
quarter STRING)")
+
+  for (country <- Seq("US", "CA", "KR")) {
+for (quarter <- 1 to 4) {
+  sql(s"ALTER TABLE sales ADD PARTITION (country = '$country', 
quarter = '$quarter')")
+}
+  }
+
+  sql("ALTER TABLE sales DROP PARTITION (country < 'KR', quarter > 
'2')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=CA/quarter=1") ::
+Row("country=CA/quarter=2") ::
+Row("country=KR/quarter=1") ::
+Row("country=KR/quarter=2") ::
+Row("country=KR/quarter=3") ::
+Row("country=KR/quarter=4") ::
+Row("country=US/quarter=1") ::
+Row("country=US/quarter=2") ::
+Row("country=US/quarter=3") ::
+Row("country=US/quarter=4") :: Nil)
+
+  sql("ALTER TABLE sales DROP PARTITION (country < 'KR'), PARTITION 
(quarter <= '1')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=KR/quarter=2") ::
+Row("country=KR/quarter=3") ::
+Row("country=KR/quarter=4") ::
+Row("country=US/quarter=2") ::
+Row("country=US/quarter=3") ::
+Row("country=US/quarter=4") :: Nil)
+
+  sql("ALTER TABLE sales DROP PARTITION (country='KR', quarter='4')")
+  sql("ALTER TABLE sales DROP PARTITION (country='US', quarter='3')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=KR/quarter=2") ::
+Row("country=KR/quarter=3") ::
+Row("country=US/quarter=2") ::
+Row("country=US/quarter=4") :: Nil)
+
+  sql("ALTER TABLE sales DROP PARTITION (quarter <= 2), PARTITION 
(quarter >= '4')")
+  checkAnswer(sql("SHOW PARTITIONS sales"),
+Row("country=KR/quarter=3") :: Nil)
+
+  val m = intercept[AnalysisException] {
+sql("ALTER TABLE sales DROP PARTITION (quarter <= 4), PARTITION 
(quarter <= '3')")
+  }.getMessage
+  // `PARTITION (quarter <= '2')` should raises exceptions because 
`PARTITION (quarter <= 4)`
--- End diff --

Incorrect comment: `PARTITION (quarter <= '2')` -> `PARTITION (quarter <= 
'3')`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/15704#discussion_r87683492
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -418,27 +419,66 @@ case class AlterTableRenamePartitionCommand(
  */
 case class AlterTableDropPartitionCommand(
 tableName: TableIdentifier,
-specs: Seq[TablePartitionSpec],
+specs: Seq[Expression],
 ifExists: Boolean,
 purge: Boolean)
-  extends RunnableCommand {
+  extends RunnableCommand with PredicateHelper {
+
+  private def isRangeComparison(expr: Expression): Boolean = {
+expr.find(e => e.isInstanceOf[BinaryComparison] && 
!e.isInstanceOf[EqualTo]).isDefined
+  }
 
   override def run(sparkSession: SparkSession): Seq[Row] = {
 val catalog = sparkSession.sessionState.catalog
 val table = catalog.getTableMetadata(tableName)
+val resolver = sparkSession.sessionState.conf.resolver
 DDLUtils.verifyAlterTableType(catalog, table, isView = false)
 DDLUtils.verifyPartitionProviderIsHive(sparkSession, table, "ALTER 
TABLE DROP PARTITION")
 
-val normalizedSpecs = specs.map { spec =>
-  PartitioningUtils.normalizePartitionSpec(
-spec,
-table.partitionColumnNames,
-table.identifier.quotedString,
-sparkSession.sessionState.conf.resolver)
+specs.foreach { expr =>
+  expr.references.foreach { attr =>
+if (!table.partitionColumnNames.exists(resolver(_, attr.name))) {
+  throw new AnalysisException(s"${attr.name} is not a valid 
partition column " +
+s"in table ${table.identifier.quotedString}.")
+}
+  }
 }
 
-catalog.dropPartitions(
-  table.identifier, normalizedSpecs, ignoreIfNotExists = ifExists, 
purge = purge)
+if (specs.exists(isRangeComparison)) {
+  if (!ifExists) {
+// Prevent query execution if one of partition specs is invalid.
+specs.foreach { spec =>
+  val partitions = 
catalog.listPartitionsByFilter(table.identifier, Seq(spec))
--- End diff --

Can we not list partitions twice? It might be time consuming task. We can 
keep listed partitions and drop it in later block.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15593: [SPARK-18060][ML] Avoid unnecessary computation for MLOR

[sethah]

Github user sethah commented on the issue:

https://github.com/apache/spark/pull/15593
  
Thanks @dbtsai!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15843: [SPARK-18274][ML][PYSPARK] Memory leak in PySpark JavaWr...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/15843
  
@jkbradley Sounds making sense more.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15593: [SPARK-18060][ML] Avoid unnecessary computation f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15593


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15593: [SPARK-18060][ML] Avoid unnecessary computation for MLOR

[dbtsai]

Github user dbtsai commented on the issue:

https://github.com/apache/spark/pull/15593
  
Thanks all for working on this PR. I merged this into master, and I'll 
create a followup task and PR to handle the abstraction together with handling 
the smoothing in the initialization of coefficients. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15856
  
@dongjoon-hyun Thanks! It has been merged. Could you close it? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15856
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15856
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68542/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15856
  
**[Test build #68542 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68542/consoleFull)**
 for PR 15856 at commit 
[`92d901b`](https://github.com/apache/spark/commit/92d901b13be0a60cfda7cd8fba4ec8bb3c0610f6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68549 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68549/consoleFull)**
 for PR 15859 at commit 
[`b60485e`](https://github.com/apache/spark/commit/b60485e7b3b76e33f46c3c017733f2eaed57f922).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68548 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68548/consoleFull)**
 for PR 15859 at commit 
[`7ae7286`](https://github.com/apache/spark/commit/7ae72863ce911e59b5a98841e80b37e51e3167ea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15853
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15853
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68545/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15853
  
**[Test build #68545 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68545/consoleFull)**
 for PR 15853 at commit 
[`dae6bf8`](https://github.com/apache/spark/commit/dae6bf804eb4972a470a6dc0b4f0aa7ef94d491b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68547 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68547/consoleFull)**
 for PR 15859 at commit 
[`0e3ef54`](https://github.com/apache/spark/commit/0e3ef545687bc9e57e001c9d55519d150e875dcb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/15859
  
LGTM pending tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15859
  
**[Test build #68546 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68546/consoleFull)**
 for PR 15859 at commit 
[`fd564e6`](https://github.com/apache/spark/commit/fd564e62d9f8563b89bd2247d25944bda355b14c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15859: Fixed state store bug

[tdas]

GitHub user tdas opened a pull request:

https://github.com/apache/spark/pull/15859

Fixed state store bug

## What changes were proposed in this pull request?

StateStore.get() causes temporary files to be created immediately, even if 
the store is not used to make updates for new version. The temp file is not 
closed as store.commit() is not called in those cases, thus keeping the output 
stream to temp file open forever.

This PR fixes it by opening the temp file only when there are updates being 
made.

## How was this patch tested?

New unit test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tdas/spark SPARK-18416

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15859.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15859


commit a9d8edf6224c25fe9b4d016cfc85d0f1d7dcfa08
Author: Tathagata Das 
Date:   2016-11-12T01:00:15Z

Fixed state store bug




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15855: [SPARK-16808][Core] History Server main page does not ho...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15855
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68540/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15855: [SPARK-16808][Core] History Server main page does not ho...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15855
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15855: [SPARK-16808][Core] History Server main page does not ho...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15855
  
**[Test build #68540 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68540/consoleFull)**
 for PR 15855 at commit 
[`844561c`](https://github.com/apache/spark/commit/844561cb5bb6127abec141eb073846dfb5aaa457).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15853
  
**[Test build #68545 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68545/consoleFull)**
 for PR 15853 at commit 
[`dae6bf8`](https://github.com/apache/spark/commit/dae6bf804eb4972a470a6dc0b4f0aa7ef94d491b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15854
  
**[Test build #68544 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68544/consoleFull)**
 for PR 15854 at commit 
[`d6a3b3f`](https://github.com/apache/spark/commit/d6a3b3f6ebad8256efe6687d6adcaf3971b90ee4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/15659#discussion_r87678827
  
--- Diff: dev/run-pip-tests-2 ---
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Stop on error
+set -e
+# Set nullglob for when we are checking existence based on globs
+shopt -s nullglob
+
+FWDIR="$(cd "`dirname $0`"/..; pwd)"
+cd "$FWDIR"
+# Some systems don't have pip or virtualenv - in those cases our tests 
won't work.
+if ! hash virtualenv 2>/dev/null; then
+  echo "Missing virtualenv skipping pip installability tests."
+  exit 0
+fi
+if ! hash pip 2>/dev/null; then
+  echo "Missing pip, skipping pip installability tests."
+  exit 0
+fi
+
+# Figure out which Python execs we should test pip installation with
+PYTHON_EXECS=()
+if hash python 2>/dev/null; then
+  # We do this since we are testing with virtualenv and the default 
virtual env python
+  # is in /usr/bin/python
+  PYTHON_EXECS+=('python')
--- End diff --

One slight oddity in AMPLab Jenkins is that `python` might actually point 
to `python3`. Given this, I think that it might be worth trying to use 
`python2` or `python2.7` or `python2.6` first and then only fall back on adding 
`python` as a last resort in order to guarantee that we're testing with a 
Python 2 environment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/15659#discussion_r87679919
  
--- Diff: dev/create-release/release-build.sh ---
@@ -187,10 +208,10 @@ if [[ "$1" == "package" ]]; then
   # We increment the Zinc port each time to avoid OOM's and other 
craziness if multiple builds
   # share the same Zinc server.
   FLAGS="-Psparkr -Phive -Phive-thriftserver -Pyarn -Pmesos"
-  make_binary_release "hadoop2.3" "-Phadoop2.3 $FLAGS" "3033" &
-  make_binary_release "hadoop2.4" "-Phadoop2.4 $FLAGS" "3034" &
-  make_binary_release "hadoop2.6" "-Phadoop2.6 $FLAGS" "3035" &
-  make_binary_release "hadoop2.7" "-Phadoop2.7 $FLAGS" "3036" &
+  make_binary_release "hadoop2.3" "-Phadoop-2.3 $FLAGS" "3033" &
+  make_binary_release "hadoop2.4" "-Phadoop-2.4 $FLAGS" "3034" &
+  make_binary_release "hadoop2.6" "-Phadoop-2.6 $FLAGS" "3035" &
+  make_binary_release "hadoop2.7" "-Phadoop-2.7 $FLAGS" "3036" &
--- End diff --

I think this is a new issue which was introduced in 
https://github.com/apache/spark/pull/14637/files#diff-01ca42240614718522afde4d4885b40dL189.
 I'd be in favor of fixing this separately. Do you mind splitting this change 
into a separate small PR which I'll merge right away?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/15659#discussion_r87678482
  
--- Diff: dev/run-pip-tests ---
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+FWDIR="$(cd "`dirname $0`"/..; pwd)"
--- End diff --

```
In dev/run-pip-tests line 21:
FWDIR="$(cd "`dirname $0`"/..; pwd)"
 ^-- SC2164: Use cd ... || exit in case cd fails.
 ^-- SC2006: Use $(..) instead of legacy `..`.
  ^-- SC2086: Double quote to prevent globbing and word 
splitting.


In dev/run-pip-tests line 22:
cd "$FWDIR"
^-- SC2164: Use cd ... || exit in case cd fails.


In dev/run-pip-tests line 26:
$FWDIR/dev/run-pip-tests-2
^-- SC2086: Double quote to prevent globbing and word splitting.


In dev/run-pip-tests line 31:
  rm -rf `cat ./virtual_env_temp_dir`
 ^-- SC2046: Quote this to prevent word splitting.
 ^-- SC2006: Use $(..) instead of legacy `..`.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/15659#discussion_r87677806
  
--- Diff: bin/beeline ---
@@ -25,7 +25,7 @@ set -o posix
 
 # Figure out if SPARK_HOME is set
 if [ -z "${SPARK_HOME}" ]; then
-  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+  source `dirname $0`/find-spark-home
--- End diff --

```
In bin/beeline line 28:
  source `dirname $0`/find-spark-home
  ^-- SC1090: Can't follow non-constant source. Use a directive to specify 
location.
 ^-- SC2046: Quote this to prevent word splitting.
 ^-- SC2006: Use $(..) instead of legacy `..`.
  ^-- SC2086: Double quote to prevent globbing and word 
splitting.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org