[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15659 **[Test build #68557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68557/consoleFull)** for PR 15659 at commit [`3345eb9`](https://github.com/apache/spark/commit/3345eb9bd3f0a165ae1d20d76e5d36e88e5512cd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13837: [SPARK-16126] [SQL] Better Error Message When using Data...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13837 **[Test build #68556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68556/consoleFull)** for PR 13837 at commit [`635046a`](https://github.com/apache/spark/commit/635046a10cc059a6ae8756fb7bc7167f5621255c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13837: [SPARK-16126] [SQL] Better Error Message When usi...
GitHub user gatorsmile reopened a pull request: https://github.com/apache/spark/pull/13837 [SPARK-16126] [SQL] Better Error Message When using DataFrameReader without `path` What changes were proposed in this pull request? When users do not specify the path in `DataFrameReader` APIs, it can get a confusing error message. For example, ``` Scala spark.read.json() ``` Error message: ``` Unable to infer schema for JSON at . It must be specified manually; ``` After the fix, the error message will be like: ``` 'path' is not specified ``` Another major goal of this PR is to add test cases for the latest changes in https://github.com/apache/spark/pull/13727. - orc read APIs - illegal format name - save API - empty path or illegal path - load API - empty path - illegal compression - fixed a test case in the existing test case `prevent all column partitioning` How was this patch tested? Test cases are added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark dfWriterAudit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13837.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13837 commit 8d021e47e9a4e95ade99d617c77ef1e17245a796 Author: gatorsmileDate: 2016-06-17T18:24:42Z test cases commit 5e4a3c666dfb767215130df1a778e5f97d438c54 Author: gatorsmile Date: 2016-06-17T19:58:56Z add test cases. commit 26437151ff0db4c0010510de047f81b1808890f4 Author: gatorsmile Date: 2016-06-17T23:48:23Z fix and test cases commit cfc0188a0baa45aef1bae6604dd10450eaafd561 Author: gatorsmile Date: 2016-06-21T01:59:02Z Merge remote-tracking branch 'upstream/master' into dfWriterAudit commit 3007fe66d03a6a40dc530c13d44c27030118a8a4 Author: gatorsmile Date: 2016-06-21T13:27:16Z more test case commit a1ae7249322c17ea09be4e968535dc115b2acb64 Author: gatorsmile Date: 2016-06-22T06:12:56Z fix test case commit 635046a10cc059a6ae8756fb7bc7167f5621255c Author: gatorsmile Date: 2016-06-22T16:04:51Z fix test case --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/15704#discussion_r87688739 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -418,27 +419,66 @@ case class AlterTableRenamePartitionCommand( */ case class AlterTableDropPartitionCommand( tableName: TableIdentifier, -specs: Seq[TablePartitionSpec], +specs: Seq[Expression], ifExists: Boolean, purge: Boolean) - extends RunnableCommand { + extends RunnableCommand with PredicateHelper { + + private def isRangeComparison(expr: Expression): Boolean = { +expr.find(e => e.isInstanceOf[BinaryComparison] && !e.isInstanceOf[EqualTo]).isDefined + } override def run(sparkSession: SparkSession): Seq[Row] = { val catalog = sparkSession.sessionState.catalog val table = catalog.getTableMetadata(tableName) +val resolver = sparkSession.sessionState.conf.resolver DDLUtils.verifyAlterTableType(catalog, table, isView = false) DDLUtils.verifyPartitionProviderIsHive(sparkSession, table, "ALTER TABLE DROP PARTITION") -val normalizedSpecs = specs.map { spec => - PartitioningUtils.normalizePartitionSpec( -spec, -table.partitionColumnNames, -table.identifier.quotedString, -sparkSession.sessionState.conf.resolver) +specs.foreach { expr => + expr.references.foreach { attr => +if (!table.partitionColumnNames.exists(resolver(_, attr.name))) { + throw new AnalysisException(s"${attr.name} is not a valid partition column " + +s"in table ${table.identifier.quotedString}.") +} + } } -catalog.dropPartitions( - table.identifier, normalizedSpecs, ignoreIfNotExists = ifExists, purge = purge) +if (specs.exists(isRangeComparison)) { + if (!ifExists) { +// Prevent query execution if one of partition specs is invalid. +specs.foreach { spec => + val partitions = catalog.listPartitionsByFilter(table.identifier, Seq(spec)) --- End diff -- Yep, correct! Thank you so much, @viirya . Then, I'll update the PR like that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/15659 So the simplest thing sounds like updating the error message, for now I'll avoid changing the maven build since that might have some unintended collateral effects and this change is currently primarily additive. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15563#discussion_r87688574 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -207,6 +207,10 @@ package object config { .booleanConf .createWithDefault(false) + private[spark] val APP_CALLER_CONTEXT = ConfigBuilder("spark.log.callerContext") --- End diff -- Thanks for the comment. Would it be a problem if we use spark.hadoop.log.callerContext? I know it gets passed into Configuration, but why would that be a problem? Is it overriding some common configuration in Hadoop? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15717: [SPARK-17910][SQL] Allow users to update the comment of ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15717 Any update on this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15854 **[Test build #68555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68555/consoleFull)** for PR 15854 at commit [`9ca806e`](https://github.com/apache/spark/commit/9ca806e2cac05610d43ffd6019c23b54ead3b40b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/15704#discussion_r87688495 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -418,27 +419,66 @@ case class AlterTableRenamePartitionCommand( */ case class AlterTableDropPartitionCommand( tableName: TableIdentifier, -specs: Seq[TablePartitionSpec], +specs: Seq[Expression], ifExists: Boolean, purge: Boolean) - extends RunnableCommand { + extends RunnableCommand with PredicateHelper { + + private def isRangeComparison(expr: Expression): Boolean = { +expr.find(e => e.isInstanceOf[BinaryComparison] && !e.isInstanceOf[EqualTo]).isDefined + } override def run(sparkSession: SparkSession): Seq[Row] = { val catalog = sparkSession.sessionState.catalog val table = catalog.getTableMetadata(tableName) +val resolver = sparkSession.sessionState.conf.resolver DDLUtils.verifyAlterTableType(catalog, table, isView = false) DDLUtils.verifyPartitionProviderIsHive(sparkSession, table, "ALTER TABLE DROP PARTITION") -val normalizedSpecs = specs.map { spec => - PartitioningUtils.normalizePartitionSpec( -spec, -table.partitionColumnNames, -table.identifier.quotedString, -sparkSession.sessionState.conf.resolver) +specs.foreach { expr => + expr.references.foreach { attr => +if (!table.partitionColumnNames.exists(resolver(_, attr.name))) { + throw new AnalysisException(s"${attr.name} is not a valid partition column " + +s"in table ${table.identifier.quotedString}.") +} + } } -catalog.dropPartitions( - table.identifier, normalizedSpecs, ignoreIfNotExists = ifExists, purge = purge) +if (specs.exists(isRangeComparison)) { + if (!ifExists) { +// Prevent query execution if one of partition specs is invalid. +specs.foreach { spec => + val partitions = catalog.listPartitionsByFilter(table.identifier, Seq(spec)) --- End diff -- I think it is good. Actually the partitions dropped in the end should be the same. The difference is only if an exception is thrown or not, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE us...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15854#discussion_r87688482 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala --- @@ -33,7 +33,9 @@ import org.apache.spark.sql.types.MetadataBuilder * ViewType is used to specify the expected view type when we want to create or replace a view in * [[CreateViewCommand]]. */ -sealed trait ViewType +sealed trait ViewType { + override def toString: String = getClass.getSimpleName.filter(_ != '$') --- End diff -- Yeah! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15659 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68554/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15659 **[Test build #68554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68554/consoleFull)** for PR 15659 at commit [`2904998`](https://github.com/apache/spark/commit/290499848cf9d65fa35e8488f11531091a014081). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...
Github user weiqingy commented on a diff in the pull request: https://github.com/apache/spark/pull/15563#discussion_r87688398 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -2578,26 +2579,38 @@ private[util] object CallerContext extends Logging { * @param taskAttemptNumber task attempt id */ private[spark] class CallerContext( - from: String, - appId: Option[String] = None, - appAttemptId: Option[String] = None, - jobId: Option[Int] = None, - stageId: Option[Int] = None, - stageAttemptId: Option[Int] = None, - taskId: Option[Long] = None, - taskAttemptNumber: Option[Int] = None) extends Logging { - - val appIdStr = if (appId.isDefined) s"_${appId.get}" else "" - val appAttemptIdStr = if (appAttemptId.isDefined) s"_${appAttemptId.get}" else "" - val jobIdStr = if (jobId.isDefined) s"_JId_${jobId.get}" else "" - val stageIdStr = if (stageId.isDefined) s"_SId_${stageId.get}" else "" - val stageAttemptIdStr = if (stageAttemptId.isDefined) s"_${stageAttemptId.get}" else "" - val taskIdStr = if (taskId.isDefined) s"_TId_${taskId.get}" else "" - val taskAttemptNumberStr = - if (taskAttemptNumber.isDefined) s"_${taskAttemptNumber.get}" else "" - - val context = "SPARK_" + from + appIdStr + appAttemptIdStr + - jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr + from: String, --- End diff -- What do you mean "off"? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...
Github user weiqingy commented on a diff in the pull request: https://github.com/apache/spark/pull/15563#discussion_r87688391 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -207,6 +207,10 @@ package object config { .booleanConf .createWithDefault(false) + private[spark] val APP_CALLER_CONTEXT = ConfigBuilder("spark.log.callerContext") --- End diff -- This is not just for Yarn, if running spark apps in standalonde mode with master and workers and reading/writing from/to hdfs, the caller context would still work on the hdfs side. (PS. we also can not use spark.hadoop prefix that will be treated as Hadoop configuration and set to `Configuration`.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15659 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68553/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15659 **[Test build #68553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68553/consoleFull)** for PR 15659 at commit [`587c0eb`](https://github.com/apache/spark/commit/587c0ebc71ff587e48ec8e71728069d155694dd2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15860 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15860 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68552/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15860 **[Test build #68552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68552/consoleFull)** for PR 15860 at commit [`f47101a`](https://github.com/apache/spark/commit/f47101aedbf6c5d9dbf2ca7f1b6456b1e867a438). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user minrk commented on the issue: https://github.com/apache/spark/pull/15659 > Any ideas on how we could follow up and support that install pattern? To go all the way to the two commands I expected to work based on Python tradition: ```bash ./build/mvn -DskipTests clean package cd python && pip install . ``` I believe you would need to copy (or hardlink or symlink-to-absolute-path) the jars into the Python directory during `build\mvn`, since `pip` copies the whole `python` directory to a temp location before running your setup.py. One step short of this would be to allow the `stage-jars` to be an explicit step, rather than a temporary staging in setup.py that cleans up after itself. Then you would have one small extra step and the full command would be: ```bash ./build/mvn cd python python setup.py stage_jars # ./stage-jars.sh, whatever you prefer pip install . ``` and the error message when jars are missing could point to the extra step. Yet another option, and the least work from what you have now, could be to put the full sequence in your missing-jars error message: > If you are installing pyspark from the spark source, you must build spark and run sdist first: > ``` > ./build/mvn -DskipTests clean package > cd python > python setup.py sdist > pip install dist/*.tar.gz > ``` I think most people wouldn't discover that without very explicit help, because they would need to know about both the temporary symlinks and pip's tempdir to work it out. Not being able to do `pip install .` is a bit odd for a Python package, as that is the standard command for installing any Python package from source (it's not `python setup.py install` anymore). But if you provide something copy/pasteable when people do try it, you are in pretty good shape. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE us...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15854#discussion_r87687899 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala --- @@ -33,7 +33,9 @@ import org.apache.spark.sql.types.MetadataBuilder * ViewType is used to specify the expected view type when we want to create or replace a view in * [[CreateViewCommand]]. */ -sealed trait ViewType +sealed trait ViewType { + override def toString: String = getClass.getSimpleName.filter(_ != '$') --- End diff -- stripSuffix? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15563#discussion_r87687846 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -2578,26 +2579,38 @@ private[util] object CallerContext extends Logging { * @param taskAttemptNumber task attempt id */ private[spark] class CallerContext( - from: String, - appId: Option[String] = None, - appAttemptId: Option[String] = None, - jobId: Option[Int] = None, - stageId: Option[Int] = None, - stageAttemptId: Option[Int] = None, - taskId: Option[Long] = None, - taskAttemptNumber: Option[Int] = None) extends Logging { - - val appIdStr = if (appId.isDefined) s"_${appId.get}" else "" - val appAttemptIdStr = if (appAttemptId.isDefined) s"_${appAttemptId.get}" else "" - val jobIdStr = if (jobId.isDefined) s"_JId_${jobId.get}" else "" - val stageIdStr = if (stageId.isDefined) s"_SId_${stageId.get}" else "" - val stageAttemptIdStr = if (stageAttemptId.isDefined) s"_${stageAttemptId.get}" else "" - val taskIdStr = if (taskId.isDefined) s"_TId_${taskId.get}" else "" - val taskAttemptNumberStr = - if (taskAttemptNumber.isDefined) s"_${taskAttemptNumber.get}" else "" - - val context = "SPARK_" + from + appIdStr + appAttemptIdStr + - jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr + from: String, --- End diff -- this is off? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/15659 Thanks @JoshRosen for taking time to review this. The factoured out fix is at https://github.com/apache/spark/pull/15860 :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15763 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68551/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15763 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15763 **[Test build #68551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68551/consoleFull)** for PR 15763 at commit [`7722503`](https://github.com/apache/spark/commit/7722503e68981f340e8a44aa91d5738b6d6a6b2a). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class AesCipher ` * `public class AesConfigMessage implements Encodable ` * `public class ByteArrayReadableChannel implements ReadableByteChannel ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15704 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15704 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68550/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15704 **[Test build #68550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68550/consoleFull)** for PR 15704 at commit [`f3f0ad5`](https://github.com/apache/spark/commit/f3f0ad546a880fa36b8f62ee67eeedd6a79bda1b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15659 **[Test build #68554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68554/consoleFull)** for PR 15659 at commit [`2904998`](https://github.com/apache/spark/commit/290499848cf9d65fa35e8488f11531091a014081). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/15659 @minrk thanks :) Any ideas on how we could follow up and support that install pattern? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15659 **[Test build #68553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68553/consoleFull)** for PR 15659 at commit [`587c0eb`](https://github.com/apache/spark/commit/587c0ebc71ff587e48ec8e71728069d155694dd2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/15659#discussion_r87686589 --- Diff: pom.xml --- @@ -26,6 +26,7 @@ org.apache.spark spark-parent_2.11 + --- End diff -- @JoshRosen so we already update this implicitly using release-tag.sh - this is just the version for dev builds. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15860: [SPARK-18418] Fix flags for make_binary_release for hado...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15860 **[Test build #68552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68552/consoleFull)** for PR 15860 at commit [`f47101a`](https://github.com/apache/spark/commit/f47101aedbf6c5d9dbf2ca7f1b6456b1e867a438). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15840: [SPARK-18398][SQL] Fix nullabilities of MapObjects and o...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/15840 I agree that this case is fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip instal...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/15659 @joshrosen Is `shellcheck` intended to be part of our standard linting? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/15659#discussion_r87686249 --- Diff: dev/create-release/release-build.sh --- @@ -187,10 +208,10 @@ if [[ "$1" == "package" ]]; then # We increment the Zinc port each time to avoid OOM's and other craziness if multiple builds # share the same Zinc server. FLAGS="-Psparkr -Phive -Phive-thriftserver -Pyarn -Pmesos" - make_binary_release "hadoop2.3" "-Phadoop2.3 $FLAGS" "3033" & - make_binary_release "hadoop2.4" "-Phadoop2.4 $FLAGS" "3034" & - make_binary_release "hadoop2.6" "-Phadoop2.6 $FLAGS" "3035" & - make_binary_release "hadoop2.7" "-Phadoop2.7 $FLAGS" "3036" & + make_binary_release "hadoop2.3" "-Phadoop-2.3 $FLAGS" "3033" & + make_binary_release "hadoop2.4" "-Phadoop-2.4 $FLAGS" "3034" & + make_binary_release "hadoop2.6" "-Phadoop-2.6 $FLAGS" "3035" & + make_binary_release "hadoop2.7" "-Phadoop-2.7 $FLAGS" "3036" & --- End diff -- Done - https://github.com/apache/spark/pull/15860 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15860: [SPARK-18418] Fix flags for make_binary_release f...
GitHub user holdenk opened a pull request: https://github.com/apache/spark/pull/15860 [SPARK-18418] Fix flags for make_binary_release for hadoop profile ## What changes were proposed in this pull request? Fix the flags used to specify the hadoop version ## How was this patch tested? Manually tested as part of https://github.com/apache/spark/pull/15659 by having the build succeed. cc @joshrosen You can merge this pull request into a Git repository by running: $ git pull https://github.com/holdenk/spark minor-fix-release-build-script Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15860.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15860 commit f47101aedbf6c5d9dbf2ca7f1b6456b1e867a438 Author: Holden KarauDate: 2016-11-12T04:09:47Z Fix flags for make_binary_release for hadoop profile - split from https://github.com/apache/spark/pull/15659 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68548/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15763: [SPARK-17348][SQL] Incorrect results from subquery trans...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15763 **[Test build #68551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68551/consoleFull)** for PR 15763 at commit [`7722503`](https://github.com/apache/spark/commit/7722503e68981f340e8a44aa91d5738b6d6a6b2a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68548/consoleFull)** for PR 15859 at commit [`7ae7286`](https://github.com/apache/spark/commit/7ae72863ce911e59b5a98841e80b37e51e3167ea). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68547/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68549/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68547/consoleFull)** for PR 15859 at commit [`0e3ef54`](https://github.com/apache/spark/commit/0e3ef545687bc9e57e001c9d55519d150e875dcb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68549/consoleFull)** for PR 15859 at commit [`b60485e`](https://github.com/apache/spark/commit/b60485e7b3b76e33f46c3c017733f2eaed57f922). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15854 @hvanhovell The PR description is updated. How about the latest change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15854 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68544/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15838: [SPARK-18396][HISTORYSERVER]"Duration" column makes sear...
Github user WangTaoTheTonic commented on the issue: https://github.com/apache/spark/pull/15838 Is it good to go? @srowen @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15854 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15854 **[Test build #68544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68544/consoleFull)** for PR 15854 at commit [`d6a3b3f`](https://github.com/apache/spark/commit/d6a3b3f6ebad8256efe6687d6adcaf3971b90ee4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15840: [SPARK-18398][SQL] Fix nullabilities of MapObjects and o...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/15840 @kiszk I have not checked all the case yet but I think the case that we need to generate else-clause doesn't match the case we discuss here. Of course we can add the method you suggested when we find there is the case and let me know if you find it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68546/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15859 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68546/consoleFull)** for PR 15859 at commit [`fd564e6`](https://github.com/apache/spark/commit/fd564e62d9f8563b89bd2247d25944bda355b14c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/13909#discussion_r87685165 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala --- @@ -43,11 +43,38 @@ trait ExpressionEvalHelper extends GeneratorDrivenPropertyChecks { protected def checkEvaluation( expression: => Expression, expected: Any, inputRow: InternalRow = EmptyRow): Unit = { -val catalystValue = CatalystTypeConverters.convertToCatalyst(expected) +// No codegen version expects GenericArrayData +val catalystValue = expected match { --- End diff -- This is still looks weird to me. Actually you need this because you replace `Seq` with `Array` in `test("CreateArray")` at `ComplexTypeSuite.scala`. Can we not change `Seq` to `Array` in `ComplexTypeSuite`? We actually can convert `Seq` to `Array` for unsafe version only in `convertToCatalystUnsafe` if we move `convertToCatalystUnsafe` to `ExpressionEvalHelper.scala`. No other place uses this `convertToCatalystUnsafe`, I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15563: [SPARK-16759][CORE] Add a configuration property to pass...
Github user weiqingy commented on the issue: https://github.com/apache/spark/pull/15563 Thanks a lot for the review, @tgravescs @mridulm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15704 **[Test build #68550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68550/consoleFull)** for PR 15704 at commit [`f3f0ad5`](https://github.com/apache/spark/commit/f3f0ad546a880fa36b8f62ee67eeedd6a79bda1b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/15704#discussion_r87684919 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -225,6 +226,111 @@ class HiveDDLSuite } } + test("SPARK-17732: Drop partitions by filter") { +withTable("sales") { + sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)") + + for (country <- Seq("US", "CA", "KR")) { +for (quarter <- 1 to 4) { + sql(s"ALTER TABLE sales ADD PARTITION (country = '$country', quarter = '$quarter')") +} + } + + sql("ALTER TABLE sales DROP PARTITION (country < 'KR', quarter > '2')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=CA/quarter=1") :: +Row("country=CA/quarter=2") :: +Row("country=KR/quarter=1") :: +Row("country=KR/quarter=2") :: +Row("country=KR/quarter=3") :: +Row("country=KR/quarter=4") :: +Row("country=US/quarter=1") :: +Row("country=US/quarter=2") :: +Row("country=US/quarter=3") :: +Row("country=US/quarter=4") :: Nil) + + sql("ALTER TABLE sales DROP PARTITION (country < 'KR'), PARTITION (quarter <= '1')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=KR/quarter=2") :: +Row("country=KR/quarter=3") :: +Row("country=KR/quarter=4") :: +Row("country=US/quarter=2") :: +Row("country=US/quarter=3") :: +Row("country=US/quarter=4") :: Nil) + + sql("ALTER TABLE sales DROP PARTITION (country='KR', quarter='4')") + sql("ALTER TABLE sales DROP PARTITION (country='US', quarter='3')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=KR/quarter=2") :: +Row("country=KR/quarter=3") :: +Row("country=US/quarter=2") :: +Row("country=US/quarter=4") :: Nil) + + sql("ALTER TABLE sales DROP PARTITION (quarter <= 2), PARTITION (quarter >= '4')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=KR/quarter=3") :: Nil) + + val m = intercept[AnalysisException] { +sql("ALTER TABLE sales DROP PARTITION (quarter <= 4), PARTITION (quarter <= '3')") + }.getMessage + // `PARTITION (quarter <= '2')` should raises exceptions because `PARTITION (quarter <= 4)` --- End diff -- Thanks. I fixed that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15563 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15563: [SPARK-16759][CORE] Add a configuration property to pass...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15563 Merging into master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15856 Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder shoul...
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/15856 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15857: [SPARK-18300][SQL] Do not apply foldable propagation wit...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15857 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15857: [SPARK-18300][SQL] Do not apply foldable propagation wit...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15857 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68543/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15857: [SPARK-18300][SQL] Do not apply foldable propagation wit...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15857 **[Test build #68543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68543/consoleFull)** for PR 15857 at commit [`d98f8f9`](https://github.com/apache/spark/commit/d98f8f930330b368fbeade07d435197b0cdd2228). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/13909#discussion_r87684528 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -421,6 +421,17 @@ object CatalystTypeConverters { case other => other } + def convertToCatalystUnsafe(a: Any): Any = a match { --- End diff -- Actually do we want to have this in `CatalystTypeConverters`? I think it should be in `ExpressionEvalHelper.scala`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15563: [SPARK-16759][CORE] Add a configuration property to pass...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15563 Looks good to me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/15704#discussion_r87683697 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -225,6 +226,111 @@ class HiveDDLSuite } } + test("SPARK-17732: Drop partitions by filter") { +withTable("sales") { + sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)") + + for (country <- Seq("US", "CA", "KR")) { +for (quarter <- 1 to 4) { + sql(s"ALTER TABLE sales ADD PARTITION (country = '$country', quarter = '$quarter')") +} + } + + sql("ALTER TABLE sales DROP PARTITION (country < 'KR', quarter > '2')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=CA/quarter=1") :: +Row("country=CA/quarter=2") :: +Row("country=KR/quarter=1") :: +Row("country=KR/quarter=2") :: +Row("country=KR/quarter=3") :: +Row("country=KR/quarter=4") :: +Row("country=US/quarter=1") :: +Row("country=US/quarter=2") :: +Row("country=US/quarter=3") :: +Row("country=US/quarter=4") :: Nil) + + sql("ALTER TABLE sales DROP PARTITION (country < 'KR'), PARTITION (quarter <= '1')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=KR/quarter=2") :: +Row("country=KR/quarter=3") :: +Row("country=KR/quarter=4") :: +Row("country=US/quarter=2") :: +Row("country=US/quarter=3") :: +Row("country=US/quarter=4") :: Nil) + + sql("ALTER TABLE sales DROP PARTITION (country='KR', quarter='4')") + sql("ALTER TABLE sales DROP PARTITION (country='US', quarter='3')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=KR/quarter=2") :: +Row("country=KR/quarter=3") :: +Row("country=US/quarter=2") :: +Row("country=US/quarter=4") :: Nil) + + sql("ALTER TABLE sales DROP PARTITION (quarter <= 2), PARTITION (quarter >= '4')") + checkAnswer(sql("SHOW PARTITIONS sales"), +Row("country=KR/quarter=3") :: Nil) + + val m = intercept[AnalysisException] { +sql("ALTER TABLE sales DROP PARTITION (quarter <= 4), PARTITION (quarter <= '3')") + }.getMessage + // `PARTITION (quarter <= '2')` should raises exceptions because `PARTITION (quarter <= 4)` --- End diff -- Incorrect comment: `PARTITION (quarter <= '2')` -> `PARTITION (quarter <= '3')`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/15704#discussion_r87683492 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -418,27 +419,66 @@ case class AlterTableRenamePartitionCommand( */ case class AlterTableDropPartitionCommand( tableName: TableIdentifier, -specs: Seq[TablePartitionSpec], +specs: Seq[Expression], ifExists: Boolean, purge: Boolean) - extends RunnableCommand { + extends RunnableCommand with PredicateHelper { + + private def isRangeComparison(expr: Expression): Boolean = { +expr.find(e => e.isInstanceOf[BinaryComparison] && !e.isInstanceOf[EqualTo]).isDefined + } override def run(sparkSession: SparkSession): Seq[Row] = { val catalog = sparkSession.sessionState.catalog val table = catalog.getTableMetadata(tableName) +val resolver = sparkSession.sessionState.conf.resolver DDLUtils.verifyAlterTableType(catalog, table, isView = false) DDLUtils.verifyPartitionProviderIsHive(sparkSession, table, "ALTER TABLE DROP PARTITION") -val normalizedSpecs = specs.map { spec => - PartitioningUtils.normalizePartitionSpec( -spec, -table.partitionColumnNames, -table.identifier.quotedString, -sparkSession.sessionState.conf.resolver) +specs.foreach { expr => + expr.references.foreach { attr => +if (!table.partitionColumnNames.exists(resolver(_, attr.name))) { + throw new AnalysisException(s"${attr.name} is not a valid partition column " + +s"in table ${table.identifier.quotedString}.") +} + } } -catalog.dropPartitions( - table.identifier, normalizedSpecs, ignoreIfNotExists = ifExists, purge = purge) +if (specs.exists(isRangeComparison)) { + if (!ifExists) { +// Prevent query execution if one of partition specs is invalid. +specs.foreach { spec => + val partitions = catalog.listPartitionsByFilter(table.identifier, Seq(spec)) --- End diff -- Can we not list partitions twice? It might be time consuming task. We can keep listed partitions and drop it in later block. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15593: [SPARK-18060][ML] Avoid unnecessary computation for MLOR
Github user sethah commented on the issue: https://github.com/apache/spark/pull/15593 Thanks @dbtsai! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15843: [SPARK-18274][ML][PYSPARK] Memory leak in PySpark JavaWr...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15843 @jkbradley Sounds making sense more. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15593: [SPARK-18060][ML] Avoid unnecessary computation f...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15593 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15593: [SPARK-18060][ML] Avoid unnecessary computation for MLOR
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/15593 Thanks all for working on this PR. I merged this into master, and I'll create a followup task and PR to handle the abstraction together with handling the smoothing in the initialization of coefficients. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15856 @dongjoon-hyun Thanks! It has been merged. Could you close it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15856 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15856 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68542/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15856: [SPARK-17982][SQL][BACKPORT-2.0] SQLBuilder should wrap ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15856 **[Test build #68542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68542/consoleFull)** for PR 15856 at commit [`92d901b`](https://github.com/apache/spark/commit/92d901b13be0a60cfda7cd8fba4ec8bb3c0610f6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68549/consoleFull)** for PR 15859 at commit [`b60485e`](https://github.com/apache/spark/commit/b60485e7b3b76e33f46c3c017733f2eaed57f922). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68548/consoleFull)** for PR 15859 at commit [`7ae7286`](https://github.com/apache/spark/commit/7ae72863ce911e59b5a98841e80b37e51e3167ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15853 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15853 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68545/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15853 **[Test build #68545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68545/consoleFull)** for PR 15853 at commit [`dae6bf8`](https://github.com/apache/spark/commit/dae6bf804eb4972a470a6dc0b4f0aa7ef94d491b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68547/consoleFull)** for PR 15859 at commit [`0e3ef54`](https://github.com/apache/spark/commit/0e3ef545687bc9e57e001c9d55519d150e875dcb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15859 LGTM pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15859: [SPARK-18416][Structured Streaming] Fixed temp file leak...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15859 **[Test build #68546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68546/consoleFull)** for PR 15859 at commit [`fd564e6`](https://github.com/apache/spark/commit/fd564e62d9f8563b89bd2247d25944bda355b14c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15859: Fixed state store bug
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/15859 Fixed state store bug ## What changes were proposed in this pull request? StateStore.get() causes temporary files to be created immediately, even if the store is not used to make updates for new version. The temp file is not closed as store.commit() is not called in those cases, thus keeping the output stream to temp file open forever. This PR fixes it by opening the temp file only when there are updates being made. ## How was this patch tested? New unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/tdas/spark SPARK-18416 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15859.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15859 commit a9d8edf6224c25fe9b4d016cfc85d0f1d7dcfa08 Author: Tathagata DasDate: 2016-11-12T01:00:15Z Fixed state store bug --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15855: [SPARK-16808][Core] History Server main page does not ho...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15855 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68540/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15855: [SPARK-16808][Core] History Server main page does not ho...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15855 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15855: [SPARK-16808][Core] History Server main page does not ho...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15855 **[Test build #68540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68540/consoleFull)** for PR 15855 at commit [`844561c`](https://github.com/apache/spark/commit/844561cb5bb6127abec141eb073846dfb5aaa457). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15853: [MINOR][SS] a minor refactor of getOffset in KafkaSource
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15853 **[Test build #68545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68545/consoleFull)** for PR 15853 at commit [`dae6bf8`](https://github.com/apache/spark/commit/dae6bf804eb4972a470a6dc0b4f0aa7ef94d491b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15854: [SPARK-18415] [SQL] Weird Plan Output when CTE used in R...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15854 **[Test build #68544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68544/consoleFull)** for PR 15854 at commit [`d6a3b3f`](https://github.com/apache/spark/commit/d6a3b3f6ebad8256efe6687d6adcaf3971b90ee4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15659#discussion_r87678827 --- Diff: dev/run-pip-tests-2 --- @@ -0,0 +1,105 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Stop on error +set -e +# Set nullglob for when we are checking existence based on globs +shopt -s nullglob + +FWDIR="$(cd "`dirname $0`"/..; pwd)" +cd "$FWDIR" +# Some systems don't have pip or virtualenv - in those cases our tests won't work. +if ! hash virtualenv 2>/dev/null; then + echo "Missing virtualenv skipping pip installability tests." + exit 0 +fi +if ! hash pip 2>/dev/null; then + echo "Missing pip, skipping pip installability tests." + exit 0 +fi + +# Figure out which Python execs we should test pip installation with +PYTHON_EXECS=() +if hash python 2>/dev/null; then + # We do this since we are testing with virtualenv and the default virtual env python + # is in /usr/bin/python + PYTHON_EXECS+=('python') --- End diff -- One slight oddity in AMPLab Jenkins is that `python` might actually point to `python3`. Given this, I think that it might be worth trying to use `python2` or `python2.7` or `python2.6` first and then only fall back on adding `python` as a last resort in order to guarantee that we're testing with a Python 2 environment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15659#discussion_r87679919 --- Diff: dev/create-release/release-build.sh --- @@ -187,10 +208,10 @@ if [[ "$1" == "package" ]]; then # We increment the Zinc port each time to avoid OOM's and other craziness if multiple builds # share the same Zinc server. FLAGS="-Psparkr -Phive -Phive-thriftserver -Pyarn -Pmesos" - make_binary_release "hadoop2.3" "-Phadoop2.3 $FLAGS" "3033" & - make_binary_release "hadoop2.4" "-Phadoop2.4 $FLAGS" "3034" & - make_binary_release "hadoop2.6" "-Phadoop2.6 $FLAGS" "3035" & - make_binary_release "hadoop2.7" "-Phadoop2.7 $FLAGS" "3036" & + make_binary_release "hadoop2.3" "-Phadoop-2.3 $FLAGS" "3033" & + make_binary_release "hadoop2.4" "-Phadoop-2.4 $FLAGS" "3034" & + make_binary_release "hadoop2.6" "-Phadoop-2.6 $FLAGS" "3035" & + make_binary_release "hadoop2.7" "-Phadoop-2.7 $FLAGS" "3036" & --- End diff -- I think this is a new issue which was introduced in https://github.com/apache/spark/pull/14637/files#diff-01ca42240614718522afde4d4885b40dL189. I'd be in favor of fixing this separately. Do you mind splitting this change into a separate small PR which I'll merge right away? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15659#discussion_r87678482 --- Diff: dev/run-pip-tests --- @@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +FWDIR="$(cd "`dirname $0`"/..; pwd)" --- End diff -- ``` In dev/run-pip-tests line 21: FWDIR="$(cd "`dirname $0`"/..; pwd)" ^-- SC2164: Use cd ... || exit in case cd fails. ^-- SC2006: Use $(..) instead of legacy `..`. ^-- SC2086: Double quote to prevent globbing and word splitting. In dev/run-pip-tests line 22: cd "$FWDIR" ^-- SC2164: Use cd ... || exit in case cd fails. In dev/run-pip-tests line 26: $FWDIR/dev/run-pip-tests-2 ^-- SC2086: Double quote to prevent globbing and word splitting. In dev/run-pip-tests line 31: rm -rf `cat ./virtual_env_temp_dir` ^-- SC2046: Quote this to prevent word splitting. ^-- SC2006: Use $(..) instead of legacy `..`. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15659: [SPARK-1267][SPARK-18129] Allow PySpark to be pip...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15659#discussion_r87677806 --- Diff: bin/beeline --- @@ -25,7 +25,7 @@ set -o posix # Figure out if SPARK_HOME is set if [ -z "${SPARK_HOME}" ]; then - export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)" + source `dirname $0`/find-spark-home --- End diff -- ``` In bin/beeline line 28: source `dirname $0`/find-spark-home ^-- SC1090: Can't follow non-constant source. Use a directive to specify location. ^-- SC2046: Quote this to prevent word splitting. ^-- SC2006: Use $(..) instead of legacy `..`. ^-- SC2086: Double quote to prevent globbing and word splitting. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org