[GitHub] spark pull request #21667: [SPARK-24691][SQL]Add new API `supportDataType` i...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199076368
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
---
@@ -156,28 +156,6 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
sql("select testType()").write.mode("overwrite").orc(orcDir)
}.getMessage
assert(msg.contains("ORC data source does not support
calendarinterval data type."))
-
- // read path
--- End diff --
Is there any write path test already?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21667 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21667 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/579/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21667: [SPARK-24691][SQL]Add new API `supportDataType` i...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199075466
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
---
@@ -152,6 +152,16 @@ trait FileFormat {
}
}
+ /**
+ * Returns whether this format supports the given [[DataType]] in
read/write path.
+ *
+ * By default all data types are supported except
[[CalendarIntervalType]] in write path.
+ */
+ def supportDataType(dataType: DataType, isReadPath: Boolean): Boolean =
dataType match {
--- End diff --
Hm, shouldn't we better whitelist them rather then blacklist?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20345 **[Test build #92466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92466/testReport)** for PR 20345 at commit [`70caacf`](https://github.com/apache/spark/commit/70caacf030b3f503f35197aef3384c4c613dd082). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21667 **[Test build #92465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92465/testReport)** for PR 21667 at commit [`7fdf603`](https://github.com/apache/spark/commit/7fdf6033b6778d06850e6ae5a0fd6e3fde76a5c2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20345 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20345 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/578/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21667 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20345 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21623 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/576/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/577/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21623 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/575/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21623 **[Test build #92463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92463/testReport)** for PR 21623 at commit [`536610e`](https://github.com/apache/spark/commit/536610e2de3addfbb570b1f17e2f854febc70080). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21666 **[Test build #92461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92461/testReport)** for PR 21666 at commit [`a4f9b1e`](https://github.com/apache/spark/commit/a4f9b1ef4531f498383b8addf4ac19773601aaef). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21611 **[Test build #92464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92464/testReport)** for PR 21611 at commit [`69c0dde`](https://github.com/apache/spark/commit/69c0dde39c13ac0333a098cbc2f40197b392409f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21659 **[Test build #92462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92462/testReport)** for PR 21659 at commit [`2378726`](https://github.com/apache/spark/commit/237872637765fe116cc35606750fcb1cd6437c0f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21652 @felixcheung @liyinan926 review pls. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/573/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/574/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21659 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21623 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21666 @shivaram --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21462: [SPARK-24428][K8S] Fix unused code
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/21462#discussion_r199071328
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
---
@@ -46,8 +46,6 @@ private[spark] class KubernetesClusterManager extends
ExternalClusterManager wit
sc: SparkContext,
masterURL: String,
scheduler: TaskScheduler): SchedulerBackend = {
-val executorSecretNamesToMountPaths =
KubernetesUtils.parsePrefixedKeyValuePairs(
- sc.conf, KUBERNETES_EXECUTOR_SECRETS_PREFIX)
--- End diff --
@mccheah @liyinan926 any more to this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21611 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21666 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92454/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92453/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21623 **[Test build #92454 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92454/testReport)** for PR 21623 at commit [`536610e`](https://github.com/apache/spark/commit/536610e2de3addfbb570b1f17e2f854febc70080). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92452/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92458/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20345 **[Test build #92458 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92458/testReport)** for PR 20345 at commit [`70caacf`](https://github.com/apache/spark/commit/70caacf030b3f503f35197aef3384c4c613dd082). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92457/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21667 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21667 **[Test build #92459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92459/testReport)** for PR 21667 at commit [`7fdf603`](https://github.com/apache/spark/commit/7fdf6033b6778d06850e6ae5a0fd6e3fde76a5c2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21659 **[Test build #92452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92452/testReport)** for PR 21659 at commit [`e00a7ae`](https://github.com/apache/spark/commit/e00a7ae0abe8a034e5859161b9b2116ae3f50bb2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92451/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92455/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21611 **[Test build #92455 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92455/testReport)** for PR 21611 at commit [`69c0dde`](https://github.com/apache/spark/commit/69c0dde39c13ac0333a098cbc2f40197b392409f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21659 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20345 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21667 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92459/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21659 **[Test build #92451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92451/testReport)** for PR 21659 at commit [`144ec68`](https://github.com/apache/spark/commit/144ec68cc14862d8e174a18989a3d7fcd01347d5). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21659: [WIP][SPARK-24530][PYTHON] Add a control to force Python...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21659 **[Test build #92453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92453/testReport)** for PR 21659 at commit [`2378726`](https://github.com/apache/spark/commit/237872637765fe116cc35606750fcb1cd6437c0f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21666 **[Test build #92457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92457/testReport)** for PR 21666 at commit [`a4f9b1e`](https://github.com/apache/spark/commit/a4f9b1ef4531f498383b8addf4ac19773601aaef). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21668: [SPARK-24690][SQL] Add a new config to control plan stat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21668 **[Test build #92460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92460/testReport)** for PR 21668 at commit [`f0db73b`](https://github.com/apache/spark/commit/f0db73bcad7741ec06560a1b6bb50bb49483b954). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/21667 @maropu @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92456/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21666 **[Test build #92456 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92456/testReport)** for PR 21666 at commit [`35ab8a2`](https://github.com/apache/spark/commit/35ab8a2359cd51b5daa4207d1067c7846ea6d249). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21667: [SPARK-24691][SQL]Add new API `supportDataType` i...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199070174
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
---
@@ -156,28 +156,6 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
sql("select testType()").write.mode("overwrite").orc(orcDir)
}.getMessage
assert(msg.contains("ORC data source does not support
calendarinterval data type."))
-
- // read path
--- End diff --
In read path, ORC should support `CalendarIntervalType` and `NullType`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21668: [SPARK-24690][SQL] Add a new config to control plan stat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21668 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21668: [SPARK-24690][SQL] Add a new config to control plan stat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21668 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/572/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21664: [SPARK-24678][CORE] NoClassDefFoundError will not be cat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21664 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92446/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21668: [SPARK-24690][SQL] Add a new config to control pl...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/21668 [SPARK-24690][SQL] Add a new config to control plan stats computation in LogicalRelation ## What changes were proposed in this pull request? This pr proposes to propose a new separate config so that `LogicalRelation` can use `rowCount` to compute data statistics in logical plans. In the master, we currently cannot enable `StarSchemaDetection.reorderStarJoins` because we need to turn off CBO to enable it but `StarSchemaDetection` internally references the `rowCount` that is used in LogicalRelation if CBO disabled. ## How was this patch tested? Added tests in `DataFrameJoinSuite`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark PlanStatsConf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21668.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21668 commit f0db73bcad7741ec06560a1b6bb50bb49483b954 Author: Takeshi Yamamuro Date: 2018-06-29T06:22:16Z Fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21668: [SPARK-24690][SQL] Add a new config to control plan stat...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21668 This comes from #20345. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21664: [SPARK-24678][CORE] NoClassDefFoundError will not be cat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21664 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21667 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21667 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/571/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21667 **[Test build #92459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92459/testReport)** for PR 21667 at commit [`7fdf603`](https://github.com/apache/spark/commit/7fdf6033b6778d06850e6ae5a0fd6e3fde76a5c2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21664: [SPARK-24678][CORE] NoClassDefFoundError will not be cat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21664 **[Test build #92446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92446/testReport)** for PR 21664 at commit [`340e07c`](https://github.com/apache/spark/commit/340e07cbc96a31b28eca52d31baf5a337566b15b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21667: [SPARK-24691][SQL]Add new API `supportDataType` i...
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/21667 [SPARK-24691][SQL]Add new API `supportDataType` in FileFormat ## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/21389, data source schema is validated before actual read/write. However, 1. Putting all the validations together in `DataSourceUtils` is tricky and hard to maintain. On second thought after review, I find that the `OrcFileFormat` in hive package is not matched, so that its validation wrong. 2. `DataSourceUtils.verifyWriteSchema` and `DataSourceUtils.verifyReadSchema` is not supposed to be called in every file format. We can move them to some upper entry. So, I propose we can add a new API `supportDataType` in FileFormat. Each file format can override the method to specify its supported/non-supported data types. ## How was this patch tested? Unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/gengliangwang/spark refactorSchemaValidate Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21667.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21667 commit 7fdf6033b6778d06850e6ae5a0fd6e3fde76a5c2 Author: Gengliang Wang Date: 2018-06-28T16:32:44Z refactor schema validation --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21623 **[Test build #92449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92449/testReport)** for PR 21623 at commit [`e959d1a`](https://github.com/apache/spark/commit/e959d1abb43a60dd8dae9ef8b716e616012b2a6c). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92449/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20345 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/570/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20345 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20345 **[Test build #92458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92458/testReport)** for PR 20345 at commit [`70caacf`](https://github.com/apache/spark/commit/70caacf030b3f503f35197aef3384c4c613dd082). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20345 I made a pr separately for adding the new config described above: https://github.com/apache/spark/compare/master...maropu:PlanStatsConf I'll file a jira and make a pr for separating discussions. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21172: [SPARK-23120][PYSPARK][ML] Add basic PMML export support...
Github user vruusmann commented on the issue: https://github.com/apache/spark/pull/21172 Here's a pointer to another PySpark-to-PMML conversion tool: https://github.com/jpmml/pyspark2pmml --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21666 **[Test build #92457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92457/testReport)** for PR 21666 at commit [`a4f9b1e`](https://github.com/apache/spark/commit/a4f9b1ef4531f498383b8addf4ac19773601aaef). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/569/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21623: [SPARK-24638][SQL] StringStartsWith support push ...
Github user stanzhai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21623#discussion_r199062132
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -378,6 +378,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_FILTER_PUSHDOWN_STRING_STARTSWITH_ENABLED =
+buildConf("spark.sql.parquet.filterPushdown.string.startsWith")
--- End diff --
It would be better if we added `.enabled` postfix.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21666 **[Test build #92456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92456/testReport)** for PR 21666 at commit [`35ab8a2`](https://github.com/apache/spark/commit/35ab8a2359cd51b5daa4207d1067c7846ea6d249). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/568/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21666: [SPARK-24535][SPARKR] fix tests on java check err...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/21666 [SPARK-24535][SPARKR] fix tests on java check error ## What changes were proposed in this pull request? ## How was this patch tested? unit test, manual You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rjavaskip Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21666.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21666 commit 6560e85dd46c677460b9879a75ec420c450542e6 Author: Felix Cheung Date: 2018-06-29T05:12:32Z skip cran check test on windows or java version fail commit 35ab8a2359cd51b5daa4207d1067c7846ea6d249 Author: Felix Cheung Date: 2018-06-29T05:13:58Z fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
