[GitHub] spark issue #23228: [MINOR][DOC] Update the condition description of seriali...
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/23228 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23272: [SPARK-26265][Core] Fix deadlock in BytesToBytesM...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/23272 [SPARK-26265][Core] Fix deadlock in BytesToBytesMap.MapIterator when locking both BytesToBytesMap.MapIterator and TaskMemoryManager ## What changes were proposed in this pull request? In `BytesToBytesMap.MapIterator.advanceToNextPage`, We will first lock this `MapIterator` and then `TaskMemoryManager` when going to free a memory page by calling `freePage`. At the same time, it is possibly that another memory consumer first locks `TaskMemoryManager` and then this `MapIterator` when it acquires memory and causes spilling on this `MapIterator`. So it ends with the `MapIterator` object holds lock to the `MapIterator` object and waits for lock on `TaskMemoryManager`, and the other consumer holds lock to `TaskMemoryManager` and waits for lock on the `MapIterator` object. To avoid deadlock here, this patch proposes to keep reference to the page to free and free it after releasing the lock of `MapIterator`. ## How was this patch tested? Added test and manually test by running the test 100 times to make sure there is no deadlock. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 SPARK-26265 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23272.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23272 commit 25e8e068047b714f27706399e1e6c03c338ac178 Author: Liang-Chi Hsieh Date: 2018-12-10T07:59:09Z Fix deadlock in BytesToBytesMap. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23262: [SPARK-26312][SQL]Converting converters in RDDCon...
Github user eatoncys commented on a diff in the pull request:
https://github.com/apache/spark/pull/23262#discussion_r240114106
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala ---
@@ -53,7 +53,7 @@ object RDDConversions {
data.mapPartitions { iterator =>
val numColumns = outputTypes.length
val mutableRow = new GenericInternalRow(numColumns)
- val converters =
outputTypes.map(CatalystTypeConverters.createToCatalystConverter)
+ val converters =
outputTypes.map(CatalystTypeConverters.createToCatalystConverter).toArray
--- End diff --
It has been modified, and the performance is the same as converting to
arrays.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23262: [SPARK-26312][SQL]Converting converters in RDDConversion...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23262 **[Test build #99899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99899/testReport)** for PR 23262 at commit [`89f3191`](https://github.com/apache/spark/commit/89f3191050d6173a7c8612f03e69f6628fde75b6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23262: [SPARK-26312][SQL]Converting converters in RDDCon...
Github user eatoncys commented on a diff in the pull request:
https://github.com/apache/spark/pull/23262#discussion_r240113822
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala ---
@@ -33,7 +33,7 @@ object RDDConversions {
data.mapPartitions { iterator =>
val numColumns = outputTypes.length
val mutableRow = new GenericInternalRow(numColumns)
- val converters =
outputTypes.map(CatalystTypeConverters.createToCatalystConverter)
+ val converters =
outputTypes.map(CatalystTypeConverters.createToCatalystConverter).toArray
--- End diff --
It is a good suggestion, and has been modified, would you like to review it
again, thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23262: [SPARK-26312][SQL]Converting converters in RDDConversion...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23262 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23271: [SPARK-26318][SQL] Enhance function merge performance in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23271 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23262: [SPARK-26312][SQL]Converting converters in RDDConversion...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23262 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5913/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23271: [SPARK-26318][SQL] Enhance function merge performance in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23271 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23271: [SPARK-26318][SQL] Enhance function merge performance in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23271 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23271: [SPARK-26318][SQL] Enhance function merge perform...
GitHub user KyleLi1985 opened a pull request:
https://github.com/apache/spark/pull/23271
[SPARK-26318][SQL] Enhance function merge performance in Row
## What changes were proposed in this pull request?
Enhance function merge performance in Row
Like do 1 time Row.merge for input
val row1 = Row("name", "work", 2314, "null", 1, "")
val row2 = Row(1, true, "name", null, "2010-10-22", 34, "location",
"situation")
val row3 = Row.fromSeq(Seq(row1,row2))
val rows = Seq(row1,row2,row3)
Row.merge(row1)
Row.merge(rows:_*)
it need 108458 millisecond and 158356 millisecond
After add this commit, it only need 24967 millisecond and 34035 millisecond
## How was this patch tested?
Unit test
Accuracy test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/KyleLi1985/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23271
commit 93c4af42d556b3779f6d56ffdf606c1132f8ef47
Author: æ亮
Date: 2018-12-10T08:05:40Z
Enhance function merge performance in Row
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23270: [SPARK-26317][BUILD] Upgrade SBT to 0.13.18
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23270 **[Test build #99898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99898/testReport)** for PR 23270 at commit [`0885c94`](https://github.com/apache/spark/commit/0885c947d3b4561df2d39f1bc9a35a06d7f0ed0c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23270: [SPARK-26317][BUILD] Upgrade SBT to 0.13.18
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23270 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5912/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23270: [SPARK-26317][BUILD] Upgrade SBT to 0.13.18
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23228: [MINOR][DOC] Update the condition description of seriali...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23228 **[Test build #99892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99892/testReport)** for PR 23228 at commit [`d5dadbf`](https://github.com/apache/spark/commit/d5dadbf30d5429c36ec3d5c2845a71c2717fd6f3). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23270: [SPARK-26317][BUILD] Upgrade SBT to 0.13.18
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23270 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99897/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23228: [MINOR][DOC] Update the condition description of seriali...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23228 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99892/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23268: [Hive][Minor] Refactor on HiveShim and Add Unit Tests
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23268 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99895/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23270: [SPARK-26317][BUILD] Upgrade SBT to 0.13.18
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23270 **[Test build #99897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99897/testReport)** for PR 23270 at commit [`0885c94`](https://github.com/apache/spark/commit/0885c947d3b4561df2d39f1bc9a35a06d7f0ed0c). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23270: [SPARK-26317][BUILD] Upgrade SBT to 0.13.18
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23270 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23270: [SPARK-26317][BUILD] Upgrade SBT to 0.13.18
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23270 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23268: [Hive][Minor] Refactor on HiveShim and Add Unit Tests
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23268 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23268: [Hive][Minor] Refactor on HiveShim and Add Unit Tests
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23268 **[Test build #99895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99895/testReport)** for PR 23268 at commit [`b68e7b1`](https://github.com/apache/spark/commit/b68e7b1421f977c7256573564b12b4cc07d31f4a). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23228: [MINOR][DOC] Update the condition description of seriali...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23228 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23132: [SPARK-26163][SQL] Parsing decimals from JSON using loca...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23132 @MaxGekk, mind fixing PR description accordingly? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming`...
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/23251 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
