[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21313 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90552/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21313 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20809: [SPARK-23667][CORE] Better scala version check
Github user gczsjdy commented on the issue: https://github.com/apache/spark/pull/20809 @vanzin Sorry but I will update it in next week, thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21313 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3176/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21313 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21313 **[Test build #90552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90552/testReport)** for PR 21313 at commit [`0c21160`](https://github.com/apache/spark/commit/0c21160751c82c4c453efea72e0553f6802f24a2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/21313 [SPARK-24187][R][SQL]Add array_join function to SparkR ## What changes were proposed in this pull request? This PR adds array_join function to SparkR ## How was this patch tested? Add unit test in test_sparkSQL.R You can merge this pull request into a Git repository by running: $ git pull https://github.com/huaxingao/spark spark-24187 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21313.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21313 commit 0c21160751c82c4c453efea72e0553f6802f24a2 Author: Huaxin GaoDate: 2018-05-13T16:07:14Z [SPARK-24187][R][SQL]Adding array_join function to SparkR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21067: [SPARK-23980][K8S] Resilient Spark driver on Kubernetes
Github user baluchicken commented on the issue: https://github.com/apache/spark/pull/21067 Rebased again to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20800#discussion_r187804796 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -511,6 +511,14 @@ class Dataset[T] private[sql]( */ def isLocal: Boolean = logicalPlan.isInstanceOf[LocalRelation] + /** + * Returns true if the `Dataset` is empty. + * + * @group basic + * @since 2.4.0 + */ + def isEmpty: Boolean = rdd.isEmpty() --- End diff -- @goungoun maybe; it depends on the Dataset. I'd follow @maropu 's lead or close this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20919: Feature/apply func to rdd
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20919 This should be closed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21301: [SPARK-24228][SQL] Fix Java lint errors
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21301#discussion_r187804661 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/partitioning/Distribution.java --- @@ -22,7 +22,8 @@ /** * An interface to represent data distribution requirement, which specifies how the records should - * be distributed among the data partitions(one {@link InputPartitionReader} outputs data for one partition). + * be distributed among the data partitions(one {@link InputPartitionReader} outputs data for one --- End diff -- Nit: space after "partitions" --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20274: [SPARK-20120][SQL][FOLLOW-UP] Better way to support spar...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20274 This seems fine, but, needs to be updated to merge again. I also don't see what it has to do with kubernetes jars though. I suspect there's a log4j classpath issue in play here, but this maybe avoids it, whatever it is. Which is good. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14291: [SPARK-16658][GRAPHX] Add EdgePartition.withVertexAttrib...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14291 I think this component is just not going to be updated further, not with a new public API. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17536: SPARK-20220 Documentation Add thrift scheduling pool con...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17536 Ping @mrchristine --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19293 **[Test build #4176 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4176/testReport)** for PR 19293 at commit [`45477fb`](https://github.com/apache/spark/commit/45477fbf00558066e3733a34e1d59ce22c192ee2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19293 @LantaoJin can you provide any more info about the bug here? but seems reasonable --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20677: Event time can't be greater then processing time. 12:21,...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20677 @deil87 if you're willing to just edit the PNG as best you can you can proceed that way. I don't think @tdas is active anymore --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21162: shaded guava is not used anywhere, seems guava is not sh...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21162 CC @vanzin but it's more complex than that as far as I know. It is still shaded. You need to read https://spark.apache.org/contributing.html --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20967: [SPARK-23857][Mesos] remove keytab check in mesos cluste...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20967 Is this one still active? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13477: [SPARK-15739][GraphX] Expose aggregateMessagesWithActive...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/13477 I doubt we'd make this a new public API in this pretty old component; I think this should be closed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17843: [Streaming] groupByKey should also disable map side comb...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17843 There's no real explanation here of why; I think this should be closed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20932: [SPARK-23812][SQL] DFS should be removed from unsupporte...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20932 It may happen to work, but I think the idea here is that we do not want to guarantee it will going forward. I think this should be closed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18903: [SPARK-21590][SS]Window start time should support negati...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18903 **[Test build #4175 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4175/testReport)** for PR 18903 at commit [`07e98e7`](https://github.com/apache/spark/commit/07e98e7d0903d72e66d0688dde11d481b41a63df). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21179: Update StreamingKMeans.scala
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21179 **[Test build #4174 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4174/testReport)** for PR 21179 at commit [`e061dce`](https://github.com/apache/spark/commit/e061dce36220809dd65df6a2da9e47c58e0811e8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21307: [SPARK-24186][R][SQL]change reverse and concat to...
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/21307#discussion_r187802112 --- Diff: R/pkg/R/functions.R --- @@ -2055,20 +2058,10 @@ setMethod("countDistinct", #' @details #' \code{concat}: Concatenates multiple input columns together into a single column. -#' If all inputs are binary, concat returns an output as binary. Otherwise, it returns as string. +#' The function works with strings, binary and compatible array columns. --- End diff -- @felixcheung I copied the sentence from scala doc. I guess it means "array columns that have data type compatible with string and binary", but I am not sure. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...
Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 Hi @HyukjinKwon It seems that @zsxwing is not really interested in this PR. Should I cancel it and close https://issues.apache.org/jira/browse/SPARK-22562 ? BTW. Why AppVeyor CI build takes so long time? The build has failed due to reaching max allowed time (1,5h) :-( --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #90551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90551/testReport)** for PR 19789 at commit [`639f11c`](https://github.com/apache/spark/commit/639f11c9fd856f58d067036f9b9131ea9efdc13d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90551/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #90551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90551/testReport)** for PR 19789 at commit [`639f11c`](https://github.com/apache/spark/commit/639f11c9fd856f58d067036f9b9131ea9efdc13d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90550/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21299 **[Test build #90550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90550/testReport)** for PR 21299 at commit [`bf8b42d`](https://github.com/apache/spark/commit/bf8b42d494d4a8f21bd08b2fd6ed531e21e4eb49). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3175/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21299 **[Test build #90550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90550/testReport)** for PR 21299 at commit [`bf8b42d`](https://github.com/apache/spark/commit/bf8b42d494d4a8f21bd08b2fd6ed531e21e4eb49). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21299 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90549/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21299 **[Test build #90549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90549/testReport)** for PR 21299 at commit [`bf8b42d`](https://github.com/apache/spark/commit/bf8b42d494d4a8f21bd08b2fd6ed531e21e4eb49). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21312 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21312 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90547/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21312 **[Test build #90547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90547/testReport)** for PR 21312 at commit [`48ef560`](https://github.com/apache/spark/commit/48ef5604212d9bb1648ea15337604709b06b8a7b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21213: [SPARK-24120] Show `Jobs` page when `jobId` is missing
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/21213 @gengliangwang Thanks. But changing URI format might make any incompatibility with other versions, thus we need to consider other aspects more. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21213: [SPARK-24120] Show `Jobs` page when `jobId` is missing
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/21213 @jongyoul @ajbozarth Personally I prefer to the current way, showing the error message: ![image](https://user-images.githubusercontent.com/1097932/39964611-9eab0790-56ba-11e8-8362-b1994db3f80d.png) If the problem is `yarn filter couldn't pass query strings`, can we consider changing the url from ``` /jobs/job?id=1 ``` to ``` /jobs/1 ``` which is more restful. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21304: Fix typo in UDF type match error message
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21304 @wzhfy wanna try to merge this one? Seems safe to merge to both master and branch-2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org