philipse created SPARK-31588:
--------------------------------
Summary: merge small files may need more common setting
Key: SPARK-31588
URL: https://issues.apache.org/jira/browse/SPARK-31588
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.4.5
Environment: spark:2.4.5
hdp:2.7
Reporter: philipse
Hi ,
SparkSql now allow us to use repartition or coalesce to manually control the
small files like the following
/*+ REPARTITION(1) */
/*+ COALESCE(1) */
But it can only be tuning case by case ,we need to decide whether we need to
use COALESCE or REPARTITION,can we try a more common way to reduce the decision
by set the target size as hive did
*Good points:*
1)we will also the new partitions number
2)with an ON-OFF parameter provided , user can close it if needed
3)the parmeter can be set at cluster level instand of user side,it will be more
easier to controll samll files.
4)greatly reduce the pressue of namenode
*Not good points:*
1)It will add a new task to calculate the target numbers by stastics the out
files.
I don't know whether we have planned this in future.
Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
