Daniel Dai created HIVE-20720:
---------------------------------
Summary: Add partition column option to JDBC handler
Key: HIVE-20720
URL: https://issues.apache.org/jira/browse/HIVE-20720
Project: Hive
Issue Type: New Feature
Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai
Currently JdbcStorageHandler does not split input in Tez. The reason is
numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in
Tez. And "mapreduce.job.maps" is not a valid param if ranger is in use. User
ends up always use 1 split.
We need to rely on this new feature if we want to support multi-splits. Here is
my proposal:
1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in
tblproperties if user want to split jdbc data source. In case
lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min
query to get this in planner. We can currently limit partitionColumn to only
numeric/date/timestamp column for simplicity
2. If partitionColumn/numPartitions are not specified, don't split input
3. Splits are equal intervals without respect to data distribution
4. There is also a "hive.sql.query.split" flag vetos the split (can be set
manually or automatically by calcite)
5. If partitionColumn is not defined, but numPartitions is defined, use
original limit/offset logic (however, don't rely on numSplit).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
