[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #89527 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89527/testReport)** for PR 21102 at commit [`2602f8e`](https://github.com/apache/spark/commit/26

[GitHub] spark pull request #21031: [SPARK-23923][SQL] Add cardinality function

2018-04-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21031#discussion_r182526092 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -407,6 +407,7 @@ object FunctionRegistry {

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2449/ Tes

[GitHub] spark issue #21099: [SPARK-23875][SQL][FollowUp] Add IndexedSeq wrapper for ...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21099 **[Test build #89512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89512/testReport)** for PR 21099 at commit [`abfae50`](https://github.com/apache/spark/commit/a

[GitHub] spark pull request #21033: [SPARK-19320][MESOS]allow specifying a hard limit...

2018-04-18 Thread susanxhuynh
Github user susanxhuynh commented on a diff in the pull request: https://github.com/apache/spark/pull/21033#discussion_r182283035 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -495,9 +500,8

[GitHub] spark issue #21099: [SPARK-23875][SQL][FollowUp] Add IndexedSeq wrapper for ...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89512/ Test PASSed. ---

[GitHub] spark issue #21099: [SPARK-23875][SQL][FollowUp] Add IndexedSeq wrapper for ...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21099 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function

2018-04-18 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21053 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-04-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r182532583 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -417,3 +418,156 @@ case class ArrayMax(chil

[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21053 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21053 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2450/ Tes

[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21053 **[Test build #89528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89528/testReport)** for PR 21053 at commit [`68e8907`](https://github.com/apache/spark/commit/68

[GitHub] spark issue #21014: [SPARK-23941][Mesos] Mesos task failed on specific spark...

2018-04-18 Thread tiboun
Github user tiboun commented on the issue: https://github.com/apache/spark/pull/21014 Hi, do I need to do something else in order for this PR to be merged ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apa

[GitHub] spark pull request #20938: [SPARK-23821][SQL] Collection function: flatten

2018-04-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/20938#discussion_r182534094 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -287,3 +289,160 @@ case class ArrayContains

[GitHub] spark issue #19222: [SPARK-10399][SPARK-23879][CORE][SQL] Introduce multiple...

2018-04-18 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19222 @rxin While I did not perf microbench for megamorphic (up to 3 `ByteArrayMemoryBlock`, `OnHeapMemoryBlock`, and `OffHeapMemoryBlock`) callsites, we confirmed that there is no performance regression in

[GitHub] spark issue #19222: [SPARK-10399][SPARK-23879][CORE][SQL] Introduce multiple...

2018-04-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19222 OK thanks please do that. Does TPC-DS even trigger 2 call sites? E.g. ByteArrayMemoryBlock and OnHeapMemoryBlock. Even there it might introduce a conditional branch after JIT that could lead to perf de

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21095 jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21095 @madanadit, thanks for following up with the PR and for your [design doc](https://docs.google.com/document/d/15-mk7UnOYNTXoF6EKaVlelWYc9DTrTXrYoodwDuAwY4/edit#heading=h.8jlem461uvwv). I had a few com

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21095 **[Test build #89529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89529/testReport)** for PR 21095 at commit [`7c1be8a`](https://github.com/apache/spark/commit/7c

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21068 **[Test build #89514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89514/testReport)** for PR 21068 at commit [`c92a090`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21068 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21068 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89514/ Test PASSed. ---

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21095 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2398/ --- --

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21095 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2451/ Tes

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21095 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC...

2018-04-18 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/21093 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21093 Thank you for review, @cloud-fan and @gatorsmile . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For a

[GitHub] spark pull request #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC...

2018-04-18 Thread dongjoon-hyun
GitHub user dongjoon-hyun reopened a pull request: https://github.com/apache/spark/pull/21093 [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4.3 ## What changes were proposed in this pull request? This PR updates Apache ORC dependencies to 1.4.3 released on February

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21093 Oops. I mistakenly click `close and comments` button. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org Fo

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21095 **[Test build #89529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89529/testReport)** for PR 21095 at commit [`7c1be8a`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21095 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21095 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89529/ Test PASSed. ---

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #89517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89517/testReport)** for PR 21102 at commit [`548a4b8`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21095 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2398/ ---

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89517/ Test FAILed. ---

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21074: [SPARK-21811][SQL] Fix the inconsistency behavior when f...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21074 **[Test build #89516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89516/testReport)** for PR 21074 at commit [`c2abce2`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21074: [SPARK-21811][SQL] Fix the inconsistency behavior when f...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21074 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89516/ Test PASSed. ---

[GitHub] spark pull request #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-18 Thread henryr
Github user henryr commented on a diff in the pull request: https://github.com/apache/spark/pull/21073#discussion_r182547477 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -115,6 +116,62 @@ case class MapValues(chi

[GitHub] spark issue #21074: [SPARK-21811][SQL] Fix the inconsistency behavior when f...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21074 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #89524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89524/testReport)** for PR 21103 at commit [`ce18ce0`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89524/ Test FAILed. ---

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20938 **[Test build #89518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89518/testReport)** for PR 20938 at commit [`9081291`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20938 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89518/ Test FAILed. ---

[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20938 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-18 Thread bersprockets
Github user bersprockets commented on a diff in the pull request: https://github.com/apache/spark/pull/21073#discussion_r182554450 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -115,6 +116,62 @@ case class MapValu

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21052 **[Test build #89521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89521/testReport)** for PR 21052 at commit [`0faa789`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21052 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21052 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89521/ Test FAILed. ---

[GitHub] spark pull request #21031: [SPARK-23923][SQL] Add cardinality function

2018-04-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21031#discussion_r182559515 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -407,6 +407,7 @@ object FunctionRegistry {

[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten

2018-04-18 Thread mn-mikke
Github user mn-mikke commented on the issue: https://github.com/apache/spark/pull/20938 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10....

2018-04-18 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21070#discussion_r182563063 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedPlainValuesReader.java --- @@ -63,115 +59,159 @@ public final

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182563748 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -62,6 +69,14 @@ case "$SPARK_K8S_CMD" in "$@"

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182564020 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala --- @@ -71,7 +77,7 @@ pr

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20280 **[Test build #89530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89530/testReport)** for PR 20280 at commit [`10bf2d0`](https://github.com/apache/spark/commit/10

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20280 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20280 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2452/ Tes

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20280 Looking at this again, I'm back to thinking this is the right fix. Based on #14469, if the `Row` objects were made with named arguments, then the intent is for elements to be looked up by field

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182566963 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile --- @@ -0,0 +1,33 @@ +# +# Licensed to the A

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182567081 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala --- @@ -71,7 +77,7 @@ priva

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182567225 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBuilder.scala --- @@ -29,9 +30,11 @@

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20280 Let me restate what I think the intended behavior of Row is: If a `Row` is made from kwargs, then the order of the fields can not be relied upon and whenever accessing data, it must be d

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182567449 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala --- @@ -88,15 +94,22 @@ pri

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread madanadit
Github user madanadit commented on the issue: https://github.com/apache/spark/pull/21095 Thanks @foxish for your feedback. As a first time contributor to Spark, I would like to limit the scope of the changes in this PR. Let me know when you're ready to review again. --- --

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21093 **[Test build #89522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89522/testReport)** for PR 21093 at commit [`fc5d976`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21093 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89522/ Test FAILed. ---

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21093 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to ...

2018-04-18 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/20280#discussion_r182569705 --- Diff: python/pyspark/sql/tests.py --- @@ -2306,18 +2306,20 @@ def test_toDF_with_schema_string(self): self.assertEqual(df.schema.simpleS

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread scottcarey
Github user scottcarey commented on the issue: https://github.com/apache/spark/pull/21070 I tested this with the addition of some changes to ParquetOptions.scala, but this alone does not allow for writing or reading zstd compressed parquet files, because it is using reflection to acqu

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21070 **[Test build #89531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89531/testReport)** for PR 21070 at commit [`5a78030`](https://github.com/apache/spark/commit/5a

[GitHub] spark issue #20936: [SPARK-23503][Structured Streaming] Enforcing sequencing...

2018-04-18 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/20936 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.a

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21070 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2453/ Tes

[GitHub] spark pull request #20936: [SPARK-23503][Structured Streaming] Enforcing seq...

2018-04-18 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20936#discussion_r182571570 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochCoordinator.scala --- @@ -137,30 +137,65 @@ private[continuo

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21093 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20280 **[Test build #89530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89530/testReport)** for PR 20280 at commit [`10bf2d0`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20280 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20280 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89530/ Test PASSed. ---

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21070 @scottcarey, Parquet will use the compressors if they are available. You can add them from an external Jar and it will work. LZ4 should also work out of the box because it is included in Hadoop 2.7.

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21061 **[Test build #89526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89526/testReport)** for PR 21061 at commit [`b1a0f7f`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21061 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21061 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89526/ Test FAILed. ---

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21093 **[Test build #89532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89532/testReport)** for PR 21093 at commit [`fc5d976`](https://github.com/apache/spark/commit/fc

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21070 **[Test build #89533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89533/testReport)** for PR 21070 at commit [`5fca3ce`](https://github.com/apache/spark/commit/5f

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21093 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21093: [SPARK-23340][SQL][BRANCH-2.3] Upgrade Apache ORC to 1.4...

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21093 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2454/ Tes

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21070 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2455/ Tes

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-18 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20280 > I think we should raise an error if __from_dict__ is set and the user tries to index using a position or a slice. I'd also like to follow up with another PR to address some of the usab

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #89523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89523/testReport)** for PR 21073 at commit [`d3d6ad6`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89523/ Test PASSed. ---

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #89527 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89527/testReport)** for PR 21102 at commit [`2602f8e`](https://github.com/apache/spark/commit/2

[GitHub] spark pull request #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-18 Thread bersprockets
Github user bersprockets commented on a diff in the pull request: https://github.com/apache/spark/pull/21073#discussion_r182575704 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -115,6 +116,62 @@ case class MapValu

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89527/ Test FAILed. ---

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-04-18 Thread scottcarey
Github user scottcarey commented on the issue: https://github.com/apache/spark/pull/21070 @rdblue The problem with zstd is that it is only in Hadoop 3.0, and dropping _that_ jar in breaks things as it is a major release. Extracting out only the ZStandardCodec from that and recom

[GitHub] spark pull request #20988: [SPARK-23877][SQL]: Use filter predicates to prun...

2018-04-18 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/20988#discussion_r182579387 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala --- @@ -129,35 +151,41 @@ case class OptimizeMetadataOnlyQu

  1   2   3   4   5   6   >