[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807959#comment-16807959 ] Ashutosh Chauhan commented on HIVE-21377: - +1 > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.01.patch, HIVE-21377.patch, > blob-cast-error.jpg > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792970#comment-16792970 ] Vihang Karajgaonkar commented on HIVE-21377: patch looks good to me. Although it isn't clear to me which patch changed the code to use extractSQLClob the partition parameter from the previous cast to string. May be we should fix extractBoolean in this patch and then create a separate JIRA to make sure direct sql path works for all the APIs for OracleDB. > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.01.patch, HIVE-21377.patch, > blob-cast-error.jpg > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789082#comment-16789082 ] Hive QA commented on HIVE-21377: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12961889/blob-cast-error.jpg {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/16438/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16438/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16438/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-03-11 01:14:55.385 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-16438/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-03-11 01:14:55.388 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 13ec7c3 HIVE-21390 : BI split strategy does not work for blob stores (Prasanth Jayachandran reviewed by Gopal V) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 13ec7c3 HIVE-21390 : BI split strategy does not work for blob stores (Prasanth Jayachandran reviewed by Gopal V) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-03-11 01:14:56.901 + rm -rf ../yetus_PreCommit-HIVE-Build-16438 + mkdir ../yetus_PreCommit-HIVE-Build-16438 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-16438 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16438/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch fatal: unrecognized input fatal: unrecognized input fatal: unrecognized input The patch does not appear to apply with p0, p1, or p2 + result=1 + '[' 1 -ne 0 ']' + rm -rf yetus_PreCommit-HIVE-Build-16438 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12961889 - PreCommit-HIVE-Build > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.01.patch, HIVE-21377.patch, > blob-cast-error.jpg > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787945#comment-16787945 ] Peter Vary commented on HIVE-21377: --- [~hibosoon]: The DirectSQL path was working for OracleDB too some time ago, so we definitely want to fix the issues there. Maybe [~vihangk1] can help us with the MetastoreDirectSqlUtils / extractSqlBoolean problems as well > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.01.patch, HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787567#comment-16787567 ] Bo commented on HIVE-21377: [~pvary] Even we pass the logic for this issue, we also see the other issue(blob-cast-error.jpg). So, could we open the jira for the implementing the DirectSQL feature with Oracle? The blog-case.error-jpg, this issues is related to MetastoreDirectSqlUtils.java {code:java} static void setPartitionParameters(String PARTITION_PARAMS, boolean convertMapNullsToEmptyStrings, PersistenceManager pm, String partIds, TreeMap partitions) throws MetaException { String queryText; queryText = "select \"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\" from " + PARTITION_PARAMS + "" + " where \"PART_ID\" in (" + partIds + ") and \"PARAM_KEY\" is not null" + " order by \"PART_ID\" asc"; loopJoinOrderedResult(pm, partitions, queryText, 0, new ApplyFunc() { @Override public void apply(Partition t, Object[] fields) { t.putToParameters((String)fields[1], fields[2])); <<< fields[2] is related to the WARN log }}); // Perform conversion of null map values for (Partition t : partitions.values()) {code} By the way, this code is already changed in last version of Hive accordingly: {code:java} public void apply(Partition t, Object[] fields) { t.putToParameters((String)fields[1], extractSqlClob(fields[2])); }}); // Perform conversion of null map values for (Partition t : partitions.values()) { t.setParameters(MetaStoreServerUtils.trimMapNulls(t.getParameters(), convertMapNullsToEmptyStrings)); {code} By the way, could we consider to open the JIRA about the implementing DirectSQL features for Oracle DB? > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.01.patch, HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786908#comment-16786908 ] Hive QA commented on HIVE-21377: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12961510/HIVE-21377.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15819 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/16386/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16386/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16386/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12961510 - PreCommit-HIVE-Build > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.01.patch, HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786865#comment-16786865 ] Hive QA commented on HIVE-21377: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 15s{color} | {color:blue} standalone-metastore/metastore-server in master has 179 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 53s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-16386/dev-support/hive-personality.sh | | git revision | master / 84f766e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-server U: standalone-metastore/metastore-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-16386/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.01.patch, HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786341#comment-16786341 ] Bo commented on HIVE-21377: [~pvary]How about this one? {code:java} // check if oracle db returned 0 or 1 for boolean value if (value instanceof Number) { try { return BooleanUtils.toBooleanObject(Integer.valueOf(((Number) value).intValue()), 1, 0, null); } catch (IllegalArgumentException iae) { // NOOP } } {code} > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785402#comment-16785402 ] Peter Vary commented on HIVE-21377: --- [~hibosoon]: Could we add unit tests so we will not have such problems later? > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784977#comment-16784977 ] Bo commented on HIVE-21377: [~pvary] I changed some code for handling the Number from Decimal to BigDecimal. > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784116#comment-16784116 ] Bo commented on HIVE-21377: [~pvary] please edit the Decimal to BigDecimal in the patch > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784085#comment-16784085 ] Bo commented on HIVE-21377: [~pvary] Thanks for your comment. They use the 12.2.0.1.0(ojdbc8.jar) for Oracle JDBC. > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((Decimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783887#comment-16783887 ] Hive QA commented on HIVE-21377: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12961037/HIVE-21377.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15818 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/16330/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16330/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16330/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12961037 - PreCommit-HIVE-Build > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((Decimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783845#comment-16783845 ] Hive QA commented on HIVE-21377: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 32s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 21s{color} | {color:blue} standalone-metastore/metastore-server in master has 179 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 16m 11s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-16330/dev-support/hive-personality.sh | | git revision | master / de61102 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-server U: standalone-metastore/metastore-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-16330/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21377.patch > > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783274#comment-16783274 ] Peter Vary commented on HIVE-21377: --- [~hibosoon]: Which version of Oracle, and which version of jdbc driver you use? Once upon a time I have been testing this codepath on our Oracle implementation (do not remember the actual version), and that seemed to be working. CC: [~karthik.manamcheri] > Using Oracle as HMS DB with DirectSQL > - > > Key: HIVE-21377 > URL: https://issues.apache.org/jira/browse/HIVE-21377 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Bo >Priority: Major > > When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log > accordingly: > {code:java} > 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore > (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due > to direct SQL failure (this is not an error): Cannot extract boolean from > column value 0 at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471) > at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462) > at > org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392) > {code} > In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean. > But Oracle return the 0 or 1 for Boolean. So we need to modify the > MetastoreDirectSqlUtils.java - [1] > So, could add this snip in this code? > {code:java} > static Boolean extractSqlBoolean(Object value) throws MetaException { > if (value == null) { > return null; > } > if (value instanceof Boolean) { > return (Boolean)value; > } > if (value instanceof Number) { // add > try { > return BooleanUtils.toBooleanObject((Decimal) value, 1, 0, null); > } catch(IllegalArugmentExeception iae){ > // NOOP > } > if (value instanceof String) { > try { > return BooleanUtils.toBooleanObject((String) value, "Y", "N", null); > } catch (IllegalArgumentException iae) { > // NOOP > } > } > throw new MetaException("Cannot extract boolean from column value " + > value); > } > {code} > [1] - > https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)