[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5267/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23103 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23103 **[Test build #99167 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99167/testReport)** for PR 23103 at commit [`39424dd`](https://github.com/apache/spark/commit/39424dd4ed796394f535e93186681fbd8f03a6f4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23103: [SPARK-26121] [Structured Streaming] Allow users ...
Github user zouzias commented on a diff in the pull request:
https://github.com/apache/spark/pull/23103#discussion_r235627877
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -538,6 +538,17 @@ private[kafka010] object KafkaSourceProvider extends
Logging {
.setIfUnset(ConsumerConfig.RECEIVE_BUFFER_CONFIG, 65536:
java.lang.Integer)
.build()
+ /**
+ * Returns a unique consumer group (group.id), allowing the user to set
the prefix of
+ * the consumer group
+ */
+ private def streamingUniqueGroupId(parameters: Map[String, String],
+ metadataPath: String): String = {
+val groupIdPrefix = parameters
+ .getOrElse("group.id.prefix", "spark-kafka-source")
--- End diff --
Makes sense, fixed in 39424dd
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23027 I like new benchmark, but I still hope we have a separate suite for in-memory benchmark. The current one looks misleading and inconsistent in this file. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23101 Welcome, @tasanuma ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22309: [SPARK-20384][SQL] Support value class in schema of Data...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22309 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user tasanuma commented on the issue: https://github.com/apache/spark/pull/23101 Thanks a lot! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23101 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22979: [SPARK-25977][SQL] Parsing decimals from CSV using local...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22979 **[Test build #99166 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99166/testReport)** for PR 22979 at commit [`2918d04`](https://github.com/apache/spark/commit/2918d04b774ed4135eab6301584261ff124335d0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23101 Thank you all. Merged to `master`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23115: [SPARK-26118][BACKPORT-2.2][WEB UI] Introducing spark.ui...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23115 This is merged to `branch-2.2`. Please close this PR. Thanks, @attilapiros . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22979: [SPARK-25977][SQL] Parsing decimals from CSV usin...
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22979#discussion_r235618859
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVExprUtils.scala
---
@@ -79,4 +83,22 @@ object CSVExprUtils {
throw new IllegalArgumentException(s"Delimiter cannot be more than
one character: $str")
}
}
+
+ def getDecimalParser(useLegacyParser: Boolean, locale: Locale): String
=> java.math.BigDecimal = {
+if (useLegacyParser) {
+ (s: String) => new BigDecimal(s.replaceAll(",", ""))
+} else {
+ val df = new DecimalFormat("", new DecimalFormatSymbols(locale))
--- End diff --
renamed
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23114 This is merged now. Please close this PR. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23114 Merged to `branch-2.3`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99163/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23111 **[Test build #99163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99163/testReport)** for PR 23111 at commit [`d3db950`](https://github.com/apache/spark/commit/d3db95082d1fc522ad5fe3b782f3b81fa39279e2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23116: [SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI redirect
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23116 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23116: [SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI redirect
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23116 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5266/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23116: [SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI redirect
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23116 **[Test build #99165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99165/testReport)** for PR 23116 at commit [`dcd517d`](https://github.com/apache/spark/commit/dcd517daaee7610cf4b75bd118baf24ba6bf40ae). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23116: [SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI re...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/23116 [SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI redirect ## What changes were proposed in this pull request? This PR is a follow-up PR of #21600 to fix the unnecessary UI redirect. ## How was this patch tested? Local verification You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-24553 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23116.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23116 commit dcd517daaee7610cf4b75bd118baf24ba6bf40ae Author: jerryshao Date: 2018-11-22T06:43:28Z Fix unnessary UI redirect --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23115: [SPARK-26118][BACKPORT-2.2][WEB UI] Introducing spark.ui...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23115 Thank you, @attilapiros . I'll do the manual test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23114 Apache Spark merge script removes '@' from the commit message in order to avoid pinging people. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23114 Oh, I got it. The PR description is wrong in this PR. The current one. ``` $ curl -H cookie http://458apiros-MBP.lan:18080/ ``` The correct one needs '@'. ``` $ curl -H @cookie http://458apiros-MBP.lan:18080/ ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5265/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23111 **[Test build #99164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99164/testReport)** for PR 23111 at commit [`973f6da`](https://github.com/apache/spark/commit/973f6da532c7cecd5e16d701066f5f3645fc51f7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23114 @attilapiros . I followed the verification steps in the PR description, but I cannot reproduce this issue at Spark 2.4.0 and 2.3.2 History Server. I'm using Mac like the following. Is there something I missed? ``` $ ls -alh cookie -rw-r--r-- 1 dongjoon staff 9.3K Nov 21 22:19 cookie $ curl --version curl 7.62.0 (x86_64-apple-darwin18.0.0) libcurl/7.62.0 SecureTransport zlib/1.2.11 Release-Date: 2018-10-31 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets ``` **Apache Spark 2.4.0** ``` $ curl --cookie cookie http://localhost:18080 | grep version % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 3165 100 31650 0 30728 0 --:--:-- --:--:-- --:--:-- 31029 2.4.0 $ curl -H cookie http://localhost:18080 | grep version % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 3165 100 31650 0 343k 0 --:--:-- --:--:-- --:--:-- 343k 2.4.0 ``` **Apache Spark 2.3.2** ``` $ curl -H cookie http://localhost:18080 | grep version % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 3082 100 30820 0 752k 0 --:--:-- --:--:-- --:--:-- 1003k 2.3.2 $ curl --cookie cookie http://localhost:18080 | grep version % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 3082 100 30820 0 1003k 0 --:--:-- --:--:-- --:--:-- 1003k 2.3.2 ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5264/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23111 **[Test build #99163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99163/testReport)** for PR 23111 at commit [`d3db950`](https://github.com/apache/spark/commit/d3db95082d1fc522ad5fe3b782f3b81fa39279e2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23111 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23109 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5263/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23109 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23109 **[Test build #99162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99162/testReport)** for PR 23109 at commit [`91193e7`](https://github.com/apache/spark/commit/91193e73c570158697150a35a8977c61d5c3f86b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23109 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23115: [SPARK-26118][BACKPORT-2.2][WEB UI] Introducing spark.ui...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23115 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99157/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23115: [SPARK-26118][BACKPORT-2.2][WEB UI] Introducing spark.ui...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23115 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23115: [SPARK-26118][BACKPORT-2.2][WEB UI] Introducing spark.ui...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23115 **[Test build #99157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99157/testReport)** for PR 23115 at commit [`9513b3f`](https://github.com/apache/spark/commit/9513b3f0de8d3f88eb488411f7c412e45689013f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99152/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23114 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99153/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23114 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23100 **[Test build #99152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99152/testReport)** for PR 23100 at commit [`40de38f`](https://github.com/apache/spark/commit/40de38f486d40a9517f9b3060bbd3cfd23d20986). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23114: [SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23114 **[Test build #99153 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99153/testReport)** for PR 23114 at commit [`01bd63b`](https://github.com/apache/spark/commit/01bd63b68f292f3e094a4a848cccb7087bd529dd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23058: [SPARK-25905][CORE] When getting a remote block, ...
Github user wypoon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23058#discussion_r235603798
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -693,9 +693,9 @@ private[spark] class BlockManager(
*/
private def getRemoteValues[T: ClassTag](blockId: BlockId):
Option[BlockResult] = {
val ct = implicitly[ClassTag[T]]
-getRemoteBytes(blockId).map { data =>
+getRemoteManagedBuffer(blockId).map { data =>
val values =
-serializerManager.dataDeserializeStream(blockId,
data.toInputStream(dispose = true))(ct)
+serializerManager.dataDeserializeStream(blockId,
data.createInputStream())(ct)
--- End diff --
I reran my system tests with a build with the assert. No issues.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99150/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99150/testReport)** for PR 23101 at commit [`f10729a`](https://github.com/apache/spark/commit/f10729a08c0444db1a814e727487deb04eb137d7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5262/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23100 **[Test build #99161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99161/testReport)** for PR 23100 at commit [`64364b9`](https://github.com/apache/spark/commit/64364b9934d191842a2a68bf1c795d14f746f8bc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 Test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/23100 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Platf...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/23043 `org.apache.spark.sql.MathFunctionsSuite.rint` seems failed due to this change. Can you check it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Platf...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23043 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99155/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Platf...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23043 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Platf...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23043 **[Test build #99155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99155/testReport)** for PR 23043 at commit [`a07e614`](https://github.com/apache/spark/commit/a07e614466d08453815e81b937003f1f5eba75ac). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99154/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder and re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23100 **[Test build #99154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99154/testReport)** for PR 23100 at commit [`64364b9`](https://github.com/apache/spark/commit/64364b9934d191842a2a68bf1c795d14f746f8bc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99159/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23111 **[Test build #99159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99159/testReport)** for PR 23111 at commit [`d3db950`](https://github.com/apache/spark/commit/d3db95082d1fc522ad5fe3b782f3b81fa39279e2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23084: [SPARK-26117][CORE][SQL]use SparkOutOfMemoryError instea...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/23084 so this PR is just code cleanup, not a real bug. thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Disable 'spark.executor.pyspark.me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23055 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99148/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Disable 'spark.executor.pyspark.me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23055 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Disable 'spark.executor.pyspark.me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23055 **[Test build #99148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99148/testReport)** for PR 23055 at commit [`fd92a4e`](https://github.com/apache/spark/commit/fd92a4e1cee9f666d7ee6f9c9fcb45367c8132a8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23043: [SPARK-26021][SQL] replace minus zero with zero i...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23043#discussion_r235596825
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala ---
@@ -723,4 +723,18 @@ class DataFrameAggregateSuite extends QueryTest with
SharedSQLContext {
"grouping expressions: [current_date(None)], value: [key: int,
value: string], " +
"type: GroupBy]"))
}
+
+ test("SPARK-26021: Double and Float 0.0/-0.0 should be equal when
grouping") {
+val colName = "i"
+val doubles = Seq(0.0d, 0.0d,
-0.0d).toDF(colName).groupBy(colName).count().collect()
--- End diff --
Without this change, `doubles` already has expected results:
```scala
Seq(0.0d, 0.0d, -0.0d).toDF(colName).groupBy(colName).count().show()
+---+-+
| i|count|
+---+-+
|0.0|3|
+---+-+
```
Do you know why?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23112 **[Test build #99160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99160/testReport)** for PR 23112 at commit [`2deac2a`](https://github.com/apache/spark/commit/2deac2af1c8079a88c6035f615af53291a79512a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23112 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23112 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99160/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23112 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23112 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5261/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23112 **[Test build #99160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99160/testReport)** for PR 23112 at commit [`2deac2a`](https://github.com/apache/spark/commit/2deac2af1c8079a88c6035f615af53291a79512a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23112 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23112 Looks okay to go. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5260/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23111 **[Test build #99159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99159/testReport)** for PR 23111 at commit [`d3db950`](https://github.com/apache/spark/commit/d3db95082d1fc522ad5fe3b782f3b81fa39279e2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99158/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23111 **[Test build #99158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99158/testReport)** for PR 23111 at commit [`29d15d8`](https://github.com/apache/spark/commit/29d15d8f22a1d08a998326c80fb5abdce60048f0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16732: [SPARK-19368][MLlib] BlockMatrix.toIndexedRowMatrix() op...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16732 **[Test build #4438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4438/testReport)** for PR 16732 at commit [`540b261`](https://github.com/apache/spark/commit/540b261cdfc34738f59fdb30511014b67ab3c828). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99156/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23111: [DO-NOT-MERGE] Increases default parallelism in PySpark ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23111 **[Test build #99156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99156/testReport)** for PR 23111 at commit [`ec0f730`](https://github.com/apache/spark/commit/ec0f7301f693811f4f782a62aae05abac33acd34). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23070: [SPARK-26099][SQL] Verification of the corrupt co...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23070 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23109 retest this please (the test failures look not persistent) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23070 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22907: [SPARK-25896][CORE][WIP] Accumulator should only be upda...
Github user carsonwang commented on the issue: https://github.com/apache/spark/pull/22907 What if there is a FetchFailure and Spark reruns some tasks in the previous succeeded shuffle map stage? That will be a new ShuffleMapStage and we will still double counting the accumulators, right? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23080: [SPARK-26108][SQL] Support custom lineSep in CSV datasou...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23080 LGTM except https://github.com/apache/spark/pull/23080/files#r235589426 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23054: [SPARK-26085][SQL] Key attribute of non-struct ty...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23054 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23109 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99147/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23109 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23080: [SPARK-26108][SQL] Support custom lineSep in CSV ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23080#discussion_r235589426
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
---
@@ -216,8 +232,13 @@ class CSVOptions(
format.setDelimiter(delimiter)
format.setQuote(quote)
format.setQuoteEscape(escape)
+lineSeparator.foreach {sep =>
+ format.setLineSeparator(sep)
+ format.setNormalizedNewline(0x00.toChar)
--- End diff --
I know we have some problems here for setting newlines more then 1
character because `setNormalizedNewline` only supports one character.
This is related with
https://github.com/apache/spark/pull/18581#issuecomment-314037750 and
https://github.com/uniVocity/univocity-parsers/issues/170
That's why I thought we can only support this for single character for now.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23109 **[Test build #99147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99147/testReport)** for PR 23109 at commit [`91193e7`](https://github.com/apache/spark/commit/91193e73c570158697150a35a8977c61d5c3f86b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23080: [SPARK-26108][SQL] Support custom lineSep in CSV ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23080#discussion_r235589448
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
---
@@ -227,7 +248,10 @@ class CSVOptions(
settings.setEmptyValue(emptyValueInRead)
settings.setMaxCharsPerColumn(maxCharsPerColumn)
settings.setUnescapedQuoteHandling(UnescapedQuoteHandling.STOP_AT_DELIMITER)
-settings.setLineSeparatorDetectionEnabled(multiLine == true)
+settings.setLineSeparatorDetectionEnabled(lineSeparatorInRead.isEmpty
&& multiLine)
+lineSeparatorInRead.foreach { _ =>
--- End diff --
nice!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23054 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
