[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 Thanks all for reviewing and thanks @tdas for merging this in! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95003/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #95003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95003/testReport)** for PR 21733 at commit [`19888ab`](https://github.com/apache/spark/commit/19888abc281d7a0689bf57e4c76bda918ad9306b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21733 Good point. That can be minor Pr. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #95003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95003/testReport)** for PR 21733 at commit [`19888ab`](https://github.com/apache/spark/commit/19888abc281d7a0689bf57e4c76bda918ad9306b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas Removed the .crc files. Thanks for reviewing! Btw, it might be good to find the way to add `.crc` files to `.gitignore` if `.crc` files are mostly useless all the times. Might worth to add another minor PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21733 LGTM. Will merge when tests pass. :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21733 This looks good!! Only one comment, please don't add the .crc files. They are useless and adds unnecessarily clutter. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas Kindly reminder. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94474/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #94474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94474/testReport)** for PR 21733 at commit [`65801a6`](https://github.com/apache/spark/commit/65801a60aa35449f45c44f5ee71d32292960cb88). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #94474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94474/testReport)** for PR 21733 at commit [`65801a6`](https://github.com/apache/spark/commit/65801a60aa35449f45c44f5ee71d32292960cb88). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94469/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #94469 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94469/testReport)** for PR 21733 at commit [`65801a6`](https://github.com/apache/spark/commit/65801a60aa35449f45c44f5ee71d32292960cb88). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #94469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94469/testReport)** for PR 21733 at commit [`65801a6`](https://github.com/apache/spark/commit/65801a60aa35449f45c44f5ee71d32292960cb88). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas Addressed review comments. Please take a look again. Thanks in advance. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94403/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #94403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94403/testReport)** for PR 21733 at commit [`e0ee04a`](https://github.com/apache/spark/commit/e0ee04af4f325db4813b8bf574c0de4cfbbbaed6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 Also added javadoc as well. Most of contents are from StateStore but I didn't copy the note to implementation for state store since it is duplicated. Please let me know if we want to add content for the parameter target state store as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #94403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94403/testReport)** for PR 21733 at commit [`e0ee04a`](https://github.com/apache/spark/commit/e0ee04af4f325db4813b8bf574c0de4cfbbbaed6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas Done running perf. test with 4 more tests: > BenchmarkMovingAggregationsListenerKeyMuchBigger rate: 16 version | input rows per second | processed rows per second | total state rows | used bytes of current state version | | | | | latest master (c9914cf) | 159877.232 | 149537.817 | 65000 | 133511303 | | patch (on top of c9914cf) | 160049.118 | 152497.945 | 65000 | 73236351 | state size: 54.854 % (reduces 45.15%) > BenchmarkMovingAggregationsListenerManyKeys rate: 12 version | input rows per second | processed rows per second | total state rows | used bytes of current state version | | | | | latest master (c9914cf) | 120266.810 | 107482.042 | 65000 | 38433719 | | patch (on top of c9914cf) | 119865.855 | 109268.772 | 65000 | 24900343 | state size: 64.787% (reduces 35.21%) > BenchmarkMovingAggregationsListenerManyValues rate: 25000 version | input rows per second | processed rows per second | total state rows | used bytes of current state version | | | | | latest master (c9914cf) | 25009.236 | 21216.126 | 9 | 77161711 (857.352 per row) | | patch (on top of c9914cf) | 25060.635 | 20774.500 | 99495 | 78230335 (786.274 per row) | state size: 91.709 % (reduces 8.29 %) > BenchmarkMovingAggregationsListenerValueMuchBigger rate: 85000 version | input rows per second | processed rows per second | total state rows | used bytes of current state version | | | | | latest master (c9914cf) | 85310.774 | 79091.271 | 1000 | 1324255 | | patch (on top of c9914cf) | 84791.761 | 79755.905 | 1000 | 1282687 | state size: 96.861 % (reduces 3.14 %) I don't find any outstanding perf. hit, and expected state size reduction is shown from all over the cases. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas Kindly reminder. I'll take the doc step when you say it's OK to go. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas I found the spare time to run performance tests though I've run only one app for now... I couldn't run the tests concurrently. Please let me know if you are not confident with the results from one app: I'll find more time to go with all test cases. Hope this number could give confident to accept the patch. > Machine info. MBP 15-inch Mid 2015 * i7 2.5Ghz (4 core) * 16GB 1600 Mhz DDR3 * SSD 512G > Test information * base commit : c9914cf (latest master branch) * patch internally rebased with base commit before testing * spark-submit options: master local[3] --driver-memory 6g * I don't run perf. test with all cores and memory: I left some spare resource for OS and background apps. > Performance test code https://github.com/HeartSaVioR/iot-trucking-app-spark-structured-streaming/blob/master/src/main/scala/com/hortonworks/spark/benchmark/BenchmarkMovingAggregationsListener.scala Please note that there're 4 more apps (big key size, big value size, many key columns, many value columns) in same repository. > Test result Both of version didn't catch up rate per seconds 20, but since processed rows per second were around 188000 I felt I don't need to adjust rate per seconds more tightly (like 185000, 19, etc...). The numbers for input rows per seconds and processed rows per second are calculated by taking average of 3 batches (38, 39, 40 respectively). The numbers regarding state are picked when total state rows went to 6. version | input rows per second | processed rows per second | total state rows | used bytes of current state version | | | | | latest master (c9914cf) | 200492.065 | 10.316 | 6 | 17,755,895 | | patch (on top of c9914cf) | 199242.598 | 188160.833 | 6 | 14,687,543 | So while two processed rows per seconds didn't show outstanding difference (under 1%), the patch reduced memory usage of state (for latest version) by 17.29 %. One thing to note is, in performance test, state is saved to the local SSD. It may give (small? trivial?) performance benefit on the patch when we set remote checkpoint directory. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93903/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93903/testReport)** for PR 21733 at commit [`b4a3807`](https://github.com/apache/spark/commit/b4a3807631cc8e12df367eeca554749fdd81a5ef). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas I've applied your review comments except documentation. (Will add WIP to the PR's title if it sounds clearer) There may be something you can add the review comments and so I'd like to work on documentation when the patch is in a shape to "ready to merge". Otherwise I'll try to find time/resource and run the performance tests again, but it might take couple of days or more to get it. Will update once I run and get new numbers. During the wait please continuous reviewing the code. It would help running the tests with latest updated patch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93903/testReport)** for PR 21733 at commit [`b4a3807`](https://github.com/apache/spark/commit/b4a3807631cc8e12df367eeca554749fdd81a5ef). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @tdas Thanks for the detailed review! I'll follow up your comments and update the patch. Btw, If my memory is right, I tried out increasing "rate" while benchmarking, but rate source itself became bottleneck. Not sure c5.xlarge is not enough or I might be missed regarding option(s). Sadly I can't run benchmark often because I don't have any dedicated machine. I would avoid running benchmark in non-dedicated machine for seeing computational limit, so paying to AWS to get dedicated instance/machine. I'll try out increasing "rate" once more soon, but please guide me if you have any suggestions to the benchmark code or approach. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93757/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93757/testReport)** for PR 21733 at commit [`e844636`](https://github.com/apache/spark/commit/e84463607bc86403c97ebf9b155b05da86a7aa73). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class MemoryStateStore extends StateStore() ` * `class StatefulOperatorsHelperSuite extends StreamTest ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93757/testReport)** for PR 21733 at commit [`e844636`](https://github.com/apache/spark/commit/e84463607bc86403c97ebf9b155b05da86a7aa73). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93324/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93324/testReport)** for PR 21733 at commit [`e844636`](https://github.com/apache/spark/commit/e84463607bc86403c97ebf9b155b05da86a7aa73). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class MemoryStateStore extends StateStore() ` * `class StatefulOperatorsHelperSuite extends StreamTest ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93315/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93318/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93318/testReport)** for PR 21733 at commit [`63dfb5d`](https://github.com/apache/spark/commit/63dfb5d2c82dfdf0a9e681fd5608f72a11dc04ed). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93315/testReport)** for PR 21733 at commit [`977428c`](https://github.com/apache/spark/commit/977428cb35a6fc0a9fa7a0ca1a51e39a94447a01). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` sealed trait StreamingAggregationStateManager extends Serializable ` * ` abstract class StreamingAggregationStateManagerBaseImpl(` * ` class StreamingAggregationStateManagerImplV1(` * ` class StreamingAggregationStateManagerImplV2(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93324/testReport)** for PR 21733 at commit [`e844636`](https://github.com/apache/spark/commit/e84463607bc86403c97ebf9b155b05da86a7aa73). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 Add tests for StatefulOperatorsHelper itself as well. (Sorry for pushing commits multiple times which trigger multiple builds. It might be ideal if older test builds are terminated once newer test build for specific PR is just launched.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 Now I'd like to propose changing default behavior to apply new path but keeping backward compatibility, so applied it to the patch. I'm still open on decision to apply it as advanced option as first approach, and happy to roll back when we decide on that way. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93315/testReport)** for PR 21733 at commit [`977428c`](https://github.com/apache/spark/commit/977428cb35a6fc0a9fa7a0ca1a51e39a94447a01). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93277/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93277/testReport)** for PR 21733 at commit [`ca198ef`](https://github.com/apache/spark/commit/ca198efa029981c568fd4712a0d368fcad4add0d). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` sealed trait StreamingAggregationStateManager extends Serializable ` * ` abstract class StreamingAggregationStateManagerBaseImpl(` * ` class StreamingAggregationStateManagerImplV1(` * ` class StreamingAggregationStateManagerImplV2(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93277/testReport)** for PR 21733 at commit [`ca198ef`](https://github.com/apache/spark/commit/ca198efa029981c568fd4712a0d368fcad4add0d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93222/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93222 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93222/testReport)** for PR 21733 at commit [`4754469`](https://github.com/apache/spark/commit/4754469ebdb36da1d3ae1234a49472716a143119). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` sealed trait StreamingAggregationStateManager extends Serializable ` * ` abstract class StreamingAggregationStateManagerBaseImpl(` * ` class StreamingAggregationStateManagerImplV1(` * ` class StreamingAggregationStateManagerImplV2(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93221/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93221/testReport)** for PR 21733 at commit [`db9d9ce`](https://github.com/apache/spark/commit/db9d9ce6dc4912672ca0af14833b5d0c239f9562). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * ` sealed trait StreamingAggregationStateManager extends Serializable ` * ` abstract class StreamingAggregationStateManagerBaseImpl(` * ` class StreamingAggregationStateManagerImplV1(` * ` class StreamingAggregationStateManagerImplV2(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93222 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93222/testReport)** for PR 21733 at commit [`4754469`](https://github.com/apache/spark/commit/4754469ebdb36da1d3ae1234a49472716a143119). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93221/testReport)** for PR 21733 at commit [`db9d9ce`](https://github.com/apache/spark/commit/db9d9ce6dc4912672ca0af14833b5d0c239f9562). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @arunmahadevan @jose-torres https://issues.apache.org/jira/browse/SPARK-24763?focusedCommentId=16541367&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16541367 I had a chance to test this patch with more kinds of use cases, and in overall enabling option shows on far or slightly better performance whereas it reduces state size according to the ratio of size of key-value pair. I'm now feeling that it would make sense to adopt new strategy to the default and use old behavior as fallback of supporting old app, but the numbers is for persuading committers and I still agree decision would be necessary from committer(s). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 I guess we would have to treat reducing state memory size to have worth to do: as described in above commit, we already optimized in HDFSBackedStateStoreProvider for reducing state store disk size (as well as network transfer) via not storing 4 bytes per each row (from both key and value). This approach would normally save more than previous optimization on value row, given key would have window information which contains two values: start and end. The main issue on this approach for me is possible perf. impact on workloads. Hopefully the workload I've covered shows even slight perf. improvement but not sure for other workloads yet. I might say we need to consider changing default behavior when I have overall good backing numbers, but in any way, I'm sure I agree that deciding from committer(s) is necessary. Would we be better to initiate mail thread in dev. mailing list? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21733 We could still save the value of the option to offsetSeqMetadata and error if it's changed. The value of using an option would just be that there's no global default; a poweruser can set the option for the queries they think would benefit without affecting all the other queries which get run. I agree it would be nice to just have some safe path allowing us to always use the new strategy. Absent that, there's an unfortunate tradeoff of reduced memory footprint vs added complexity. I think we ultimately need a committer to decide whether that's worth it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 @arunmahadevan I'm actually in favor of changing default behavior, just not 100% sure the result would be promising for exhaustive use cases. I might need to prepare more kinds of key/value pair (key size bigger than value size, key size smaller than value size, key size equals to value size, what else I'm missing here?) and run some tests and back it up with new numbers. Btw, as you commented, there seems two approaches to identify the old and new format: > looking at the fields in the row Actually I tried to do it before (via checking count of fields in value row, since this patch reduces the count of fields in value row), and soon realized I can't do it because HDFSBackedStateStoreProvider relies on provided keySchema and valueSchema when serializing / deserializing rows, not leveraging UnsafeRow's serialization/deserialization mechanism (writeExternal/readExternal or write/read via Kyro), so it will just show undefined behavior if the schema doesn't match with actual rows, and we can't verify this. Current approach saves cost to write/read two additional integers with sacrificing the way to verify the rows. If we would want to add the feature, state migration should be happened. > introducing a row version to differentiate old vs new We could do this via applying same approach in #21739 so this is valid, but query with old state format should do state migration (not easy to do since it should be done against multiple versions of states), or continue relying on old state format. @jose-torres Could you please take a look at @arunmahadevan 's comment as well as this comment and comment yours? Thanks in advance! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21733 @HeartSaVioR , the results looks promising. I am wondering if theres a way to make this default option than introducing new configs. Since this is internal details anyway theres no need to expose any config if we can identify the old vs new format by looking at the fields in the row or by introducing a row version to differentiate old vs new. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92791/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92791/testReport)** for PR 21733 at commit [`bb5f672`](https://github.com/apache/spark/commit/bb5f672d141cf797c75c0bc23fc0cd285a3898bb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92791/testReport)** for PR 21733 at commit [`bb5f672`](https://github.com/apache/spark/commit/bb5f672d141cf797c75c0bc23fc0cd285a3898bb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92755/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92755/testReport)** for PR 21733 at commit [`bb5f672`](https://github.com/apache/spark/commit/bb5f672d141cf797c75c0bc23fc0cd285a3898bb). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92755/testReport)** for PR 21733 at commit [`bb5f672`](https://github.com/apache/spark/commit/bb5f672d141cf797c75c0bc23fc0cd285a3898bb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92738/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92738/testReport)** for PR 21733 at commit [`bb5f672`](https://github.com/apache/spark/commit/bb5f672d141cf797c75c0bc23fc0cd285a3898bb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92738/testReport)** for PR 21733 at commit [`bb5f672`](https://github.com/apache/spark/commit/bb5f672d141cf797c75c0bc23fc0cd285a3898bb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92735/testReport)** for PR 21733 at commit [`89a30ab`](https://github.com/apache/spark/commit/89a30ab22a5af6adec9917626dcb69906f40d3c9). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92735/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92735/testReport)** for PR 21733 at commit [`89a30ab`](https://github.com/apache/spark/commit/89a30ab22a5af6adec9917626dcb69906f40d3c9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92734/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92734/testReport)** for PR 21733 at commit [`2a9cc49`](https://github.com/apache/spark/commit/2a9cc496bb7f832b75b0090ef9a612f4fbc0f206). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #92734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92734/testReport)** for PR 21733 at commit [`2a9cc49`](https://github.com/apache/spark/commit/2a9cc496bb7f832b75b0090ef9a612f4fbc0f206). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21733 cc. @tdas @zsxwing @jose-torres @jerryshao @arunmahadevan @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
