[
https://issues.apache.org/jira/browse/SPARK-16357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zeweichen updated SPARK-16357:
------------------------------
Description: During our performance test of Spark shuffle RPC encryption
using 3DES on Sparksql, we found that throughput test of 2 stream lasts 2.68X
time than power test, which is much larger than it should be. Q30 cost 0.5 hour
in stream1, but q30 last 2.5 hours in stream 0 which is much longer than it
should be. This caused that queries in stream1 go ahead much faster than
stream0. So it works like single stream and can not fully use CPU resource.
(was: During our performance test of Spark shuffle RPC encryption using 3DES on
Sparksql, we found that throughput test of 2 stream lasts 2.68X time than power
test, which is much larger than it should be. Reason is that q30 cost 0.5 hour
in stream1, but q30 last 2.5 hours in stream 0 which is much longer than it
should be. )
> For Spark shuffle RPC encryption using 3DES on Sparksql, throughput test of 2
> stream lasts 2.68X time than power test.
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-16357
> URL: https://issues.apache.org/jira/browse/SPARK-16357
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.1
> Environment: Apache Spark 1.6.1 with local patch for supporting Q7
> subquery feature
> Apache Spark 1.6.1 with Spark shuffle RPC encryption
> BigBench 1.0.1
> Reporter: zeweichen
> Priority: Minor
> Fix For: 2.1.0
>
>
> During our performance test of Spark shuffle RPC encryption using 3DES on
> Sparksql, we found that throughput test of 2 stream lasts 2.68X time than
> power test, which is much larger than it should be. Q30 cost 0.5 hour in
> stream1, but q30 last 2.5 hours in stream 0 which is much longer than it
> should be. This caused that queries in stream1 go ahead much faster than
> stream0. So it works like single stream and can not fully use CPU resource.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
