[GitHub] spark pull request #20705: [SPARK-23553][TESTS] Tests should not assume the ...

2018-03-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20705#discussion_r171729865 --- Diff: python/pyspark/sql/readwriter.py --- @@ -147,8 +147,8 @@ def load(self, path=None, format=None, schema=None, **options):

[GitHub] spark issue #20698: [SPARK-23541][SS] Allow Kafka source to read data with g...

2018-03-01 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/20698 cc @brkyvz --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spar

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171732015 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -199,10 +179,10 @@ private[kafka010] class K

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171732183 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -199,10 +179,10 @@ private[kafka010] class

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20706 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20706 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87853/ Test FAILed. ---

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20706 **[Test build #87853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87853/testReport)** for PR 20706 at commit [`c239d96`](https://github.com/apache/spark/commit/c

[GitHub] spark pull request #20710: [SPARK-23559][SS] Add epoch ID to DataWriterFacto...

2018-03-01 Thread jose-torres
GitHub user jose-torres opened a pull request: https://github.com/apache/spark/pull/20710 [SPARK-23559][SS] Add epoch ID to DataWriterFactory. ## What changes were proposed in this pull request? Add an epoch ID argument to DataWriterFactory for use in streaming. As detailed

[GitHub] spark issue #20710: [SPARK-23559][SS] Add epoch ID to DataWriterFactory.

2018-03-01 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/20710 @tdas @rdblue @cloud-fan I haven't forgotten that we need a design doc before finalization; SPARK-23556 tracks that. --- -

[GitHub] spark issue #20710: [SPARK-23559][SS] Add epoch ID to DataWriterFactory.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20710 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20710: [SPARK-23559][SS] Add epoch ID to DataWriterFactory.

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20710 **[Test build #87862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87862/testReport)** for PR 20710 at commit [`5bbd497`](https://github.com/apache/spark/commit/5b

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20704 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.ap

[GitHub] spark issue #20710: [SPARK-23559][SS] Add epoch ID to DataWriterFactory.

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20710 **[Test build #87862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87862/testReport)** for PR 20710 at commit [`5bbd497`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #20710: [SPARK-23559][SS] Add epoch ID to DataWriterFactory.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20710 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87862/ Test FAILed. ---

[GitHub] spark issue #20710: [SPARK-23559][SS] Add epoch ID to DataWriterFactory.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20710 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2018-03-01 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r171736235 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -50,12 +52,10 @@ // These are only updated by readExte

[GitHub] spark issue #20639: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20639 **[Test build #87856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87856/testReport)** for PR 20639 at commit [`55aa8bc`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #20639: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20639 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87856/ Test FAILed. ---

[GitHub] spark issue #20639: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20639 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20704 Thank you for review, @jerryshao ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comman

[GitHub] spark pull request #20682: [SPARK-23522][Python] always use sys.exit over bu...

2018-03-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20682#discussion_r171738290 --- Diff: python/pyspark/heapq3.py --- @@ -883,7 +883,7 @@ def nlargest(n, iterable, key=None): if __name__ == "__main__": -impo

[GitHub] spark pull request #20682: [SPARK-23522][Python] always use sys.exit over bu...

2018-03-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20682#discussion_r171738396 --- Diff: python/pyspark/ml/stat.py --- @@ -15,6 +15,7 @@ # limitations under the License. # +import sys --- End diff --

[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain

2018-03-01 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/20692 My intent is just to advocate for clear feedback on the content of PRs. Good to hear your confidence in @mgaido91, and if he wants to work on a better explain, that's great too. --- --

[GitHub] spark pull request #20682: [SPARK-23522][Python] always use sys.exit over bu...

2018-03-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20682#discussion_r171738655 --- Diff: python/pyspark/mllib/random.py --- @@ -19,6 +19,8 @@ Python package for random data generation. """ +import sys + -

[GitHub] spark pull request #20682: [SPARK-23522][Python] always use sys.exit over bu...

2018-03-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20682#discussion_r171738603 --- Diff: python/pyspark/ml/tuning.py --- @@ -15,6 +15,7 @@ # limitations under the License. # import itertools +import sys --- En

[GitHub] spark pull request #20682: [SPARK-23522][Python] always use sys.exit over bu...

2018-03-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20682#discussion_r171738756 --- Diff: python/pyspark/mllib/regression.py --- @@ -15,6 +15,7 @@ # limitations under the License. # +import sys --- End diff --

[GitHub] spark pull request #20682: [SPARK-23522][Python] always use sys.exit over bu...

2018-03-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20682#discussion_r171738801 --- Diff: python/pyspark/sql/conf.py --- @@ -15,6 +15,7 @@ # limitations under the License. # +import sys --- End diff --

[GitHub] spark issue #20682: [SPARK-23522][Python] always use sys.exit over builtin e...

2018-03-01 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20682 I think `dev/run_tests` doesn't actually test most of the changes (checking doctest failure) here because it needs a test failure. Can you check one of them with a manual test failure for sure?

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171733038 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala --- @@ -0,0 +1,105 @@ +/* + * Licens

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171732729 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -320,28 +300,39 @@ private[kafka010] class

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171733181 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala --- @@ -0,0 +1,105 @@ +/* + * Licens

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171732779 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -370,8 +361,14 @@ private[kafka010] class

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171733516 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala --- @@ -0,0 +1,105 @@ +/* + * Licens

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20681 **[Test build #87858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87858/testReport)** for PR 20681 at commit [`d0eacc2`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87858/ Test FAILed. ---

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20703: [SPARK-19185][SS] Make Kafka consumer cache confi...

2018-03-01 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/20703#discussion_r171741027 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -76,6 +76,10 @@ private[kafka010] c

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171741765 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala --- @@ -0,0 +1,105 @@ +/* + * Licensed

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20706 **[Test build #87855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87855/testReport)** for PR 20706 at commit [`d026fff`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20706 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20706 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87855/ Test FAILed. ---

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20704 **[Test build #87854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87854/testReport)** for PR 20704 at commit [`dbb5ae5`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20704 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87854/ Test PASSed. ---

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20706 I'm almost sure there's a bug somewhere for that flaky test, but if there isn't I'll file one. --- - To unsubscribe, e-mail: revi

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20704 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20706 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h..

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20706 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1211/ Tes

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20706 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20472 **[Test build #87864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87864/testReport)** for PR 20472 at commit [`656abef`](https://github.com/apache/spark/commit/65

[GitHub] spark issue #20706: [SPARK-23550][core] Cleanup `Utils`.

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20706 **[Test build #87863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87863/testReport)** for PR 20706 at commit [`d026fff`](https://github.com/apache/spark/commit/d0

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20704 Hmm, I guess it was just luck that this didn't trigger the deps check, since that jar is checked for a specific version (2.7.3 in the case of hadoop2.7). LGTM, merging to master / 2.3. --

[GitHub] spark pull request #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-cl...

2018-03-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20704 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20472 **[Test build #87864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87864/testReport)** for PR 20472 at commit [`656abef`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20472 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20472 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87864/ Test FAILed. ---

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16578 **[Test build #87859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87859/testReport)** for PR 16578 at commit [`27737a0`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16578 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87859/ Test PASSed. ---

[GitHub] spark issue #20618: [SPARK-23329][SQL] Fix documentation of trigonometric fu...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20618 **[Test build #87860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87860/testReport)** for PR 20618 at commit [`10afda5`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20704 Thank you for review and merging, @vanzin . We generated both `spark-deps-hadoop-2.6` and `spark-deps-hadoop-2.7` with the following. ``` ./dev/test-dependencies.sh --replace-m

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16578 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20618: [SPARK-23329][SQL] Fix documentation of trigonometric fu...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20618 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87860/ Test FAILed. ---

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2018-03-01 Thread Gauravshah
Github user Gauravshah commented on the issue: https://github.com/apache/spark/pull/16578 @marmbrus can we target it for 2.4 ? need help on reviews. Been in waiting state for very long --- - To unsubscribe, e-mail:

[GitHub] spark issue #20618: [SPARK-23329][SQL] Fix documentation of trigonometric fu...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20618 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2018-03-01 Thread Gauravshah
Github user Gauravshah commented on the issue: https://github.com/apache/spark/pull/16578 we have back-ported it to 2.2, on production by an average it has saved us at least 2x time. --- - To unsubscribe, e-mail: re

[GitHub] spark issue #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-co...

2018-03-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20704 Yeah, I'm just wondering why that didn't happen in the dependency:tree output in your description. Anyway, not really important to figure that out. ---

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20472 **[Test build #87865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87865/testReport)** for PR 20472 at commit [`fea3aad`](https://github.com/apache/spark/commit/fe

[GitHub] spark pull request #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-01 Thread gerashegalov
Github user gerashegalov commented on a diff in the pull request: https://github.com/apache/spark/pull/20327#discussion_r171747629 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -79,6 +80,19 @@ private[spark] class Appli

[GitHub] spark issue #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20327 **[Test build #87866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87866/testReport)** for PR 20327 at commit [`a674863`](https://github.com/apache/spark/commit/a6

[GitHub] spark pull request #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-01 Thread gerashegalov
Github user gerashegalov commented on a diff in the pull request: https://github.com/apache/spark/pull/20327#discussion_r171748638 --- Diff: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala --- @@ -123,6 +123,10 @@ class YarnClusterSuite ex

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-03-01 Thread liufengdb
Github user liufengdb commented on the issue: https://github.com/apache/spark/pull/20681 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark pull request #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-01 Thread gerashegalov
Github user gerashegalov commented on a diff in the pull request: https://github.com/apache/spark/pull/20327#discussion_r171749008 --- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala --- @@ -126,7 +126,7 @@ private[spark] abstract class WebUI( def bind(): Unit = {

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-01 Thread gerashegalov
Github user gerashegalov commented on a diff in the pull request: https://github.com/apache/spark/pull/20327#discussion_r171749148 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -79,6 +80,19 @@ private[spark] class Appli

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20681 **[Test build #87867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87867/testReport)** for PR 20681 at commit [`d0eacc2`](https://github.com/apache/spark/commit/d0

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1212/ Tes

[GitHub] spark pull request #20702: [SPARK-23547][SQL]Cleanup the .pipeout file when ...

2018-03-01 Thread zuotingbing
Github user zuotingbing commented on a diff in the pull request: https://github.com/apache/spark/pull/20702#discussion_r171749296 --- Diff: sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java --- @@ -665,6 +667,25 @@ public void close() thr

[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain

2018-03-01 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20692 My 2 cents: a data source may have a lot of columns, and a column may be a complex struct type. So embedding the type in the plan may make the plan unreadable. If I'm debugging and see a

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171750437 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala --- @@ -0,0 +1,105 @@ +/* + * Licensed

[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain

2018-03-01 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20692 BTW I think we do need a story to improve the explain result. The current explain works pretty well with simple queries. But for complex queries, that a single plan node across many lines, it's re

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171750580 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala --- @@ -0,0 +1,105 @@ +/* + * Licensed

[GitHub] spark pull request #20698: [SPARK-23541][SS] Allow Kafka source to read data...

2018-03-01 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20698#discussion_r171750758 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala --- @@ -0,0 +1,105 @@ +/* + * Licensed

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2018-03-01 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r171750857 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -99,8 +99,15 @@ public static UTF8String fromBytes(byte[] bytes, in

[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...

2018-03-01 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r171750940 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -67,6 +67,8 @@ sparkSession <- if (windows_with_hadoop()) { sparkR.session(master = spark

[GitHub] spark pull request #20711: [SPARKR][DOC] fix link in vignettes

2018-03-01 Thread felixcheung
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/20711 [SPARKR][DOC] fix link in vignettes ## What changes were proposed in this pull request? Fix doc link that was changed in 2.3 @shivaram You can merge this pull request into a

[GitHub] spark issue #20711: [SPARKR][DOC] fix link in vignettes

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20711 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1213/ Tes

[GitHub] spark issue #20698: [SPARK-23541][SS] Allow Kafka source to read data with g...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20698 **[Test build #87870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87870/testReport)** for PR 20698 at commit [`3eae3f1`](https://github.com/apache/spark/commit/3e

[GitHub] spark issue #20711: [SPARKR][DOC] fix link in vignettes

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20711 **[Test build #87869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87869/testReport)** for PR 20711 at commit [`d09a804`](https://github.com/apache/spark/commit/d0

[GitHub] spark issue #20698: [SPARK-23541][SS] Allow Kafka source to read data with g...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20698 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19108: [SPARK-21898][ML] Feature parity for KolmogorovSmirnovTe...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19108 **[Test build #87868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87868/testReport)** for PR 19108 at commit [`aa9772e`](https://github.com/apache/spark/commit/aa

[GitHub] spark issue #20698: [SPARK-23541][SS] Allow Kafka source to read data with g...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20698 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1214/ Tes

[GitHub] spark issue #20711: [SPARKR][DOC] fix link in vignettes

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20711 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19108: [SPARK-21898][ML] Feature parity for KolmogorovSmirnovTe...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19108 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19108: [SPARK-21898][ML] Feature parity for KolmogorovSmirnovTe...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19108 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1215/ Tes

[GitHub] spark issue #20711: [SPARKR][DOC] fix link in vignettes

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20711 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1216/ Tes

[GitHub] spark issue #20711: [SPARKR][DOC] fix link in vignettes

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20711 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20711: [SPARKR][DOC] fix link in vignettes

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20711 **[Test build #87871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87871/testReport)** for PR 20711 at commit [`17a116e`](https://github.com/apache/spark/commit/17

[GitHub] spark issue #20111: [SPARK-22883][ML][TEST] Streaming tests for spark.ml.fea...

2018-03-01 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/20111 Merging with master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20472 **[Test build #87865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87865/testReport)** for PR 20472 at commit [`fea3aad`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20472 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle perform...

2018-03-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20472 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87865/ Test PASSed. ---

<    1   2   3   4   5   6   7   >