caican00 commented on a change in pull request #35764:
URL: https://github.com/apache/spark/pull/35764#discussion_r826605591
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
##########
@@ -111,6 +111,9 @@ class JDBCOptions(
// the number of partitions
val numPartitions = parameters.get(JDBC_NUM_PARTITIONS).map(_.toInt)
+ // the default number of partitions
+ val defaultNumPartitions = parameters.getOrElse(DEFAULT_NUM_PARTITIONS,
"10").toInt
Review comment:
> What is the difference between `JDBC_NUM_PARTITIONS`?
@wangyum
1. partitionColumn, lowerBound, upperBound and numPartitions must be
specified together. If an unreasonable numPartitions was specified by users,
such as 1, the parallelism is still very small.
2. therefore, we(jdbc) should rezoning partition nums using another config,
not JDBC_NUM_PARTITIONS(it is specified by users and its value maybe very small)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
