GitHub user lvdongr opened a pull request:
https://github.com/apache/spark/pull/20356
[SPARK-23185][SQL] Make the configuration "spark.default.parallelism" can
be changed on each SQL session to decrease empty files
## What changes were proposed in this pull request?
Make the configuration "spark.default.parallelism" can be changed on each
SQL session to decrease empty files.
When execute "insert into ... values ...", many empty files will be
generated.We can change the configuration "spark.default.parallelism" to
decrease the number of empty files.But there are many occasions that we want to
chang the configuration during each session so as not to influence other sql
sentences, like we may use thrift server to excute many sql sentences on a SQL
session.
## How was this patch tested?
unit tests, manual tests
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lvdongr/spark SPARK-23185
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20356.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20356
----
commit 01af8ce69afeade8bb034c6965de0f3738f12fd5
Author: lvdongr <lv.dongdong@...>
Date: 2017-03-08T04:09:40Z
[SPARK-19863][DStream] Whether or not use CachedKafkaConsumer need to be
configured, when you use DirectKafkaInputDStream to connect the kafka in a
Spark Streaming application has been successfully created.
commit b6daeec664d757999e257e56fed3844db51515e2
Author: lvdongr <lv.dongdong@...>
Date: 2017-03-11T06:35:57Z
Merge remote-tracking branch 'apache/master'
commit e0e47b1da93b90210e44abc6e90655d3028555ec
Author: lvdongr <lv.dongdong@...>
Date: 2017-04-12T07:20:01Z
Merge remote-tracking branch 'apache/master'
commit f4ab88111c5b8e9700eacc1acfa3858aed45124e
Author: lvdongr <lv.dongdong@...>
Date: 2017-07-27T01:54:56Z
isklakldsng branch 'apache/master'
commit 463e570f9e05f785834e27bd535cfbb3b7cb7dfb
Author: lvdongr <lv.dongdong@...>
Date: 2017-07-27T12:09:47Z
Merge remote-tracking branch 'apache/master'
commit 0e1b7f6d8e436ca243f78e3cbf064f591557b6c0
Author: lvdongr <lv.dongdong@...>
Date: 2017-07-28T01:34:48Z
Merge remote-tracking branch 'apache/master'
commit 9a9972125ae8f7d90f5567f5b561f2c0ca16cfe7
Author: lvdongr <lv.dongdong@...>
Date: 2017-07-28T02:50:23Z
refresh the master branch for kafkaconsumer
commit 637900b576b8c4d9e04a808a078e481a99751d03
Author: lvdongr <lv.dongdong@...>
Date: 2017-07-31T03:08:29Z
Merge remote-tracking branch 'apache/master'
commit 04aafed076cb704a100eb7dc45b5cfda6438193b
Author: lvdongr <lv.dongdong@...>
Date: 2017-08-17T11:41:58Z
Merge remote-tracking branch 'apache/master'
commit 9f90ab5356b74dfc63dc9c80ff336ef2c2847e72
Author: root <root@...>
Date: 2017-11-10T03:32:54Z
Merge branch 'master' of https://github.com/apache/spark
commit 8b94711b7fb6cfa72aa06d9e009b73b73ccda36f
Author: root <root@...>
Date: 2017-11-13T00:56:22Z
Merge branch 'master' of https://github.com/apache/spark
commit 70699e3d80d853f7105d967544378c5c342d2ce6
Author: 10171592 <lv.dongdong@...>
Date: 2017-12-07T03:13:24Z
Merge remote-tracking branch 'apache/master'
commit 9e7c0c7d0f8bae30bc07abbedf4c110ec82f1cf3
Author: root <root@...>
Date: 2017-12-07T05:48:04Z
Merge remote-tracking branch 'apache/master'
commit 393730415bcebdef125364be3eb3a64320cac3c9
Author: root <root@...>
Date: 2018-01-09T03:16:36Z
Merge branch 'master' of https://github.com/lvdongr/spark
commit 5db407930d4802b6075036961688192a3039d95a
Author: root <root@...>
Date: 2018-01-09T03:30:43Z
Merge branch 'master' of https://github.com/apache/spark
commit 46672ddaf53b9ed1e97e404753fa14bd3406821a
Author: 10171592 <lv.dongdong@...>
Date: 2018-01-22T08:20:47Z
Merge remote-tracking branch 'apache/master'
commit 884eaee9f2d7782bceae73806da9b65f1119977e
Author: 10171592 <lv.dongdong@...>
Date: 2018-01-22T09:30:14Z
Merge branch 'master' of https://github.com/lvdongr/spark
commit 49641920727f426e88ac32a9c1381f7876eaf7c9
Author: 10171592 <lv.dongdong@...>
Date: 2018-01-23T02:57:54Z
Merge remote-tracking branch 'apache/master'
commit e1aeff8c0cb1358d0c77b0e729ecdfd1a07313dc
Author: lvdongr <lv.dongdong@...>
Date: 2018-01-23T03:18:28Z
[SPARK-23185][SQL] Make the configuration "spark.default.parallelism" can
be changed on each SQL session to decrease empty files
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
