[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) ptest failures are definitely unrelated as .9.patch and .10.patch show different failures. Patch pushed to master. Thanks [~jcamachorodriguez] for review! > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20720.1.patch, HIVE-20720.10.patch, > HIVE-20720.2.patch, HIVE-20720.3.patch, HIVE-20720.4.patch, > HIVE-20720.5.patch, HIVE-20720.6.patch, HIVE-20720.7.patch, > HIVE-20720.8.patch, HIVE-20720.9.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.10.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.10.patch, > HIVE-20720.2.patch, HIVE-20720.3.patch, HIVE-20720.4.patch, > HIVE-20720.5.patch, HIVE-20720.6.patch, HIVE-20720.7.patch, > HIVE-20720.8.patch, HIVE-20720.9.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.9.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch, HIVE-20720.9.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.8.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.7.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.6.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, HIVE-20720.6.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.5.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.4.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: (was: HIVE-20720.4.patch) > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.4.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.3.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.2.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Description: Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. SQLStdAuth) is in use. User ends up always use 1 split. We need to rely on this new feature if we want to support multi-splits. Here is my proposal: 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min query to get this in planner. We can currently limit partitionColumn to only numeric/date/timestamp column for simplicity 2. If partitionColumn/numPartitions are not specified, don't split input 3. Splits are equal intervals without respect to data distribution 4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or automatically by calcite) 5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset logic (however, don't rely on numSplit). was: Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param if ranger is in use. User ends up always use 1 split. We need to rely on this new feature if we want to support multi-splits. Here is my proposal: 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min query to get this in planner. We can currently limit partitionColumn to only numeric/date/timestamp column for simplicity 2. If partitionColumn/numPartitions are not specified, don't split input 3. Splits are equal intervals without respect to data distribution 4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or automatically by calcite) 5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset logic (however, don't rely on numSplit). > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Status: Patch Available (was: Open) Preliminary patch. More tests to come. > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if ranger is in use. > User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.1.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if ranger is in use. > User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
