[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-10 Thread ssonker
Github user ssonker commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194305723 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -111,6 +113,23 @@ object DateTimeUtils { compu

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21427 @icexelloss @HyukjinKwon It is always simple to deprecate the confs in the release of Spark 3.0. Let us make it configurable in the next release which is Spark 2.4. --- --

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21202 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21202 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91647/ Test FAILed. ---

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21202 **[Test build #91647 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91647/testReport)** for PR 21202 at commit [`3e410cd`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21495 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3888/ Tes

[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21495 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6

2018-06-10 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21495 I decided to remove the hack I put to get the Spark UI consistent since this hack will bring into more problems. @som-snytt Is it possible to move the `printWelcome` and `splash.start()` t

[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21495 **[Test build #91650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91650/testReport)** for PR 21495 at commit [`82ca5f6`](https://github.com/apache/spark/commit/82

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91645/ Test PASSed. ---

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21504 **[Test build #91645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91645/testReport)** for PR 21504 at commit [`421e16b`](https://github.com/apache/spark/commit/4

[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-10 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194299485 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -111,6 +113,23 @@ object DateTimeUtils { compute

[GitHub] spark issue #21497: [SPARK-24466][SS] Fix TextSocketMicroBatchReader to be c...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21497 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91646/ Test PASSed. ---

[GitHub] spark issue #21497: [SPARK-24466][SS] Fix TextSocketMicroBatchReader to be c...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21497 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21505: [SPARK-24457][SQL] Improving performance of stringToTime...

2018-06-10 Thread ssonker
Github user ssonker commented on the issue: https://github.com/apache/spark/pull/21505 @kiszk I've added a benchmark according to your recommendation, please review. --- - To unsubscribe, e-mail: reviews-unsubscr...

[GitHub] spark issue #21497: [SPARK-24466][SS] Fix TextSocketMicroBatchReader to be c...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21497 **[Test build #91646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91646/testReport)** for PR 21497 at commit [`d069dd0`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21520: [SPARK-24505][SQL] Forbidding string interpolation in Co...

2018-06-10 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21520 For 1, I agree with you that it is not good to introduce many APIs at first. On the other hand, it would be good to prepare only a few APIs that are frequently used, not to prepare many APIs. It make

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r194298446 --- Diff: python/pyspark/ml/feature.py --- @@ -2610,6 +2610,9 @@ def setParams(self, inputCol=None, outputCol=None, stopWords=None, caseSensitive

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-10 Thread dongjinleekr
Github user dongjinleekr commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r194296251 --- Diff: python/pyspark/ml/feature.py --- @@ -2610,6 +2610,9 @@ def setParams(self, inputCol=None, outputCol=None, stopWords=None, caseSensitive

[GitHub] spark pull request #21514: [SPARK-22860] [Core] - hide key password from lin...

2018-06-10 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21514#discussion_r194296152 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala --- @@ -100,7 +100,7 @@ private[spark] class Standalo

[GitHub] spark issue #21514: [SPARK-22860] [Core] - hide key password from linux ps l...

2018-06-10 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21514 Have you tried the config "spark.redaction.regex" ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For ad

[GitHub] spark issue #21506: [SPARK-24485][SS] Measure and log elapsed time for files...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21506 **[Test build #91649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91649/testReport)** for PR 21506 at commit [`3d0e23f`](https://github.com/apache/spark/commit/3d

[GitHub] spark pull request #21506: [SPARK-24485][SS] Measure and log elapsed time fo...

2018-06-10 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/21506#discussion_r194295068 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -280,38 +278,49 @@ private[s

[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194294961 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -125,7 +125,6 @@ object DateTimeUtils { .getO

[GitHub] spark issue #21501: [SPARK-15064][ML] Locale support in StopWordsRemover

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21501 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21501: [SPARK-15064][ML] Locale support in StopWordsRemover

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21501 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91648/ Test FAILed. ---

[GitHub] spark issue #21501: [SPARK-15064][ML] Locale support in StopWordsRemover

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21501 **[Test build #91648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91648/testReport)** for PR 21501 at commit [`b4249c3`](https://github.com/apache/spark/commit/b

[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...

2018-06-10 Thread som-snytt
Github user som-snytt commented on a diff in the pull request: https://github.com/apache/spark/pull/21495#discussion_r194294485 --- Diff: repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala --- @@ -21,8 +21,22 @@ import scala.collection.mutable im

[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-10 Thread ssonker
Github user ssonker commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194294182 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -125,7 +125,6 @@ object DateTimeUtils { .get

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r194293510 --- Diff: python/pyspark/ml/feature.py --- @@ -2610,6 +2610,9 @@ def setParams(self, inputCol=None, outputCol=None, stopWords=None, caseSensitive

[GitHub] spark pull request #21506: [SPARK-24485][SS] Measure and log elapsed time fo...

2018-06-10 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/21506#discussion_r194293481 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -280,38 +278,49 @@ private[s

[GitHub] spark pull request #21506: [SPARK-24485][SS] Measure and log elapsed time fo...

2018-06-10 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/21506#discussion_r194293251 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -280,38 +278,49 @@ private[s

[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-10 Thread ssonker
Github user ssonker commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194292883 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -111,6 +113,23 @@ object DateTimeUtils { compu

[GitHub] spark pull request #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20838#discussion_r194292645 --- Diff: dev/merge_spark_pr.py --- @@ -39,6 +39,9 @@ except ImportError: JIRA_IMPORTED = False +if sys.version_info[0] >= 3:

[GitHub] spark pull request #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20838#discussion_r194292662 --- Diff: dev/create-release/releaseutils.py --- @@ -49,6 +49,9 @@ print("Install using 'sudo pip install unidecode'") sys.exit(-1)

[GitHub] spark pull request #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20838#discussion_r194292586 --- Diff: python/pyspark/sql/conf.py --- @@ -59,7 +62,7 @@ def unset(self, key): def _checkType(self, obj, identifier): """As

[GitHub] spark pull request #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20838#discussion_r194292548 --- Diff: python/pyspark/streaming/dstream.py --- @@ -23,6 +23,8 @@ if sys.version < "3": from itertools import imap as map, ifilter

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194292067 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +354,68 @@ def show(self, n=20, truncate=True, vertical=False): else:

[GitHub] spark issue #21501: [SPARK-15064][ML] Locale support in StopWordsRemover

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21501 **[Test build #91648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91648/testReport)** for PR 21501 at commit [`b4249c3`](https://github.com/apache/spark/commit/b4

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-10 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193953345 --- Diff: core/src/main/scala/org/apache/spark/util/RpcUtils.scala --- @@ -44,7 +44,7 @@ private[spark] object RpcUtils { /** Returns the default S

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-10 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193953432 --- Diff: core/src/main/scala/org/apache/spark/barrier/BarrierRDD.scala --- @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194287915 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +354,68 @@ def show(self, n=20, truncate=True, vertical=False): else:

[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...

2018-06-10 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21495#discussion_r194287473 --- Diff: repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala --- @@ -21,8 +21,22 @@ import scala.collection.mutable im

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91644/ Test PASSed. ---

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20838 **[Test build #91644 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91644/testReport)** for PR 20838 at commit [`fd4d922`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21520: [SPARK-24505][SQL] Forbidding string interpolation in Co...

2018-06-10 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21520 > 1. We are seeing many inline prefix with a few typical patterns. > Can we introduce new APIs to avoid repetations of adding inline, for example JavaCode.className(Class[_]): JavaCode for the fir

[GitHub] spark issue #21520: [SPARK-24505][SQL] Forbidding string interpolation in Co...

2018-06-10 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21520 @kiszk @mgaido91 Thanks for your comment! > What do you think about starting doing the needed changes in smaller PRs which focus only on specific part and forbidding the string interpolation

[GitHub] spark issue #21067: [SPARK-23980][K8S] Resilient Spark driver on Kubernetes

2018-06-10 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21067 any update? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21202 **[Test build #91647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91647/testReport)** for PR 21202 at commit [`3e410cd`](https://github.com/apache/spark/commit/3e

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-06-10 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21202 @devaraj-kavali could you rebase this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comm

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-06-10 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21202 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #21486: [SPARK-24387][Core] Heartbeat-timeout executor is added ...

2018-06-10 Thread lirui-apache
Github user lirui-apache commented on the issue: https://github.com/apache/spark/pull/21486 cc @vanzin @andrewor14 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19755: [SPARK-22524][SQL] Subquery shows reused on UI SQ...

2018-06-10 Thread gczsjdy
Github user gczsjdy closed the pull request at: https://github.com/apache/spark/pull/19755 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21504 **[Test build #91645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91645/testReport)** for PR 21504 at commit [`421e16b`](https://github.com/apache/spark/commit/42

[GitHub] spark issue #21497: [SPARK-24466][SS] Fix TextSocketMicroBatchReader to be c...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21497 **[Test build #91646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91646/testReport)** for PR 21497 at commit [`d069dd0`](https://github.com/apache/spark/commit/d0

[GitHub] spark issue #21497: [SPARK-24466][SS] Fix TextSocketMicroBatchReader to be c...

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21497 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21504 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark pull request #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop ite...

2018-06-10 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21467 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21467 Merged to master. @e-dorigatti, it has some conflicts in branch-2.3 too. Mind if I ask to open a backporting PR again to reduce the difference between master and branch-2.3? ---

[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6

2018-06-10 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21495 Having issues tested with latest patch: ``` Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullComplet

[GitHub] spark issue #21481: [SPARK-24452][SQL][Core] Avoid possible overflow in int ...

2018-06-10 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21481 Thank you for your comment. I will create another PR for integrating findBugs/SpotBugs into maven. --- - To unsubscribe, e-mail: r

[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-06-10 Thread edwinalu
Github user edwinalu commented on the issue: https://github.com/apache/spark/pull/21221 @squito , I'm modifying ExecutorMetrics to take in the metrics array -- this will be easier for tests where we pass in set values, and seems fine for the actual code. It will check that the length

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 It's at least not trivial as much as Scaia side's. I am okay but please make sure what case we will allow by this configuration. --- ---

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194278100 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +354,68 @@ def show(self, n=20, truncate=True, vertical=False): else:

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194277542 --- Diff: docs/configuration.md --- @@ -456,6 +456,33 @@ Apart from these, the following properties are also available, and may be useful from J

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21370 @xuanyuanking Thanks for your contributions! Test coverage is the most critical when we refactor the existing code and add new features. Hopefully, when you submit new PRs in the future, could yo

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194277082 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -3209,6 +3222,19 @@ class Dataset[T] private[sql]( } }

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194276795 --- Diff: docs/configuration.md --- @@ -456,6 +456,33 @@ Apart from these, the following properties are also available, and may be useful from J

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194276735 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +354,68 @@ def show(self, n=20, truncate=True, vertical=False): else:

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194276557 --- Diff: docs/configuration.md --- @@ -456,6 +456,33 @@ Apart from these, the following properties are also available, and may be useful from J

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194276329 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +354,68 @@ def show(self, n=20, truncate=True, vertical=False): else:

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194276298 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -3209,6 +3222,19 @@ class Dataset[T] private[sql]( } }

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194276179 --- Diff: python/pyspark/sql/tests.py --- @@ -3074,6 +3074,36 @@ def test_checking_csv_header(self): finally: shutil.rmtree(

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21370 @xuanyuanking @HyukjinKwon Sorry for the delay. Super busy in the week of Spark summit. Will carefully review this PR today or tomorrow. --- ---

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194275282 --- Diff: docs/configuration.md --- @@ -456,6 +456,33 @@ Apart from these, the following properties are also available, and may be useful from J

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194275288 --- Diff: docs/configuration.md --- @@ -456,6 +456,33 @@ Apart from these, the following properties are also available, and may be useful from J

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-10 Thread bkrieger
Github user bkrieger commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194274619 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,32 @@ class Analyzer( expr.fi

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-10 Thread bkrieger
Github user bkrieger commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194274604 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,32 @@ class Analyzer( expr.fi

[GitHub] spark issue #21452: [MINOR][CORE] Log committer class used by HadoopMapRedCo...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21452 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21452: [MINOR][CORE] Log committer class used by HadoopMapRedCo...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91640/ Test PASSed. ---

[GitHub] spark issue #21452: [MINOR][CORE] Log committer class used by HadoopMapRedCo...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21452 **[Test build #91640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91640/testReport)** for PR 21452 at commit [`9881d9c`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21524: [SPARK-24212][ML][doc] Add the example and user guide fo...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21524 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21524: [SPARK-24212][ML][doc] Add the example and user guide fo...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21524 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21524: [SPARK-24212][ML][doc] Add the example and user guide fo...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21524 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21524: [SPARK-24212][ML][doc] Add the example and user g...

2018-06-10 Thread tengpeng
GitHub user tengpeng opened a pull request: https://github.com/apache/spark/pull/21524 [SPARK-24212][ML][doc] Add the example and user guide for ML PrefixSpan ## What changes were proposed in this pull request? There are no example and user guide for ML PrefixSpan (not MLlib

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21438 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21438 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91641/ Test PASSed. ---

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21438 **[Test build #91641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91641/testReport)** for PR 21438 at commit [`eb87d2d`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194273874 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,32 @@ class Analyzer( expr.

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194273780 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,32 @@ class Analyzer( expr.

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21045 **[Test build #91643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91643/testReport)** for PR 21045 at commit [`d8f3dea`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21045 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91643/ Test FAILed. ---

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21045 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20838 **[Test build #91644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91644/testReport)** for PR 20838 at commit [`fd4d922`](https://github.com/apache/spark/commit/fd

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21045 **[Test build #91643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91643/testReport)** for PR 21045 at commit [`d8f3dea`](https://github.com/apache/spark/commit/d8

[GitHub] spark issue #16006: [SPARK-18580] [DStreams] [external/kafka-0-10] Use spark...

2018-06-10 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/16006 #19431 was merged, thanks for your work. This PR should probably be closed. --- - To unsubscribe, e-mail: reviews-unsubscr...@

[GitHub] spark issue #21481: [SPARK-24452][SQL][Core] Avoid possible overflow in int ...

2018-06-10 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/21481 Let's merge this as-is and do the build improvements in a separate PR. That's important because we may want to backport the overflow fix to maintenance branches and may want to do so independent o

[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-10 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194268734 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -111,6 +113,23 @@ object DateTimeUtils { compute

  1   2   >