[jira] [Updated] (HIVE-1734) Implement map_keys() and map_values() UDFs
[ https://issues.apache.org/jira/browse/HIVE-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-1734: -- Assignee: Carl Steinbach (was: Mac Yang) Implement map_keys() and map_values() UDFs -- Key: HIVE-1734 URL: https://issues.apache.org/jira/browse/HIVE-1734 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.6.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-1734-mapudf.1.patch.txt, MapKeys.java, MapValues.java Implement the following UDFs: array map_keys(map) and array map_values(map) map_keys() takes a map as input and returns an array consisting of the key values in the supplied map. Similarly, map_values() takes a map as input and returns an array containing the map value fields. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1772) optimize join followed by a groupby
[ https://issues.apache.org/jira/browse/HIVE-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-1772: Attachment: HIVE-1772.1.patch initial patch.. dependent to HIVE-2339 optimize join followed by a groupby --- Key: HIVE-1772 URL: https://issues.apache.org/jira/browse/HIVE-1772 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Attachments: HIVE-1772.1.patch explain SELECT x.key, count(1) FROM src1 x JOIN src y ON (x.key = y.key) group by x.key; STAGE DEPENDENCIES: Stage-1 is a root stage Stage-2 depends on stages: Stage-1 Stage-0 is a root stage The above query issues 2 map-reduce jobs. The first MR job performs the join, whereas the second MR performs the group by. Since the data is already sorted, the group by can be performed in the reducer of the join itself. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1631) JDBC driver returns wrong precision, scale, or column size for some data types
[ https://issues.apache.org/jira/browse/HIVE-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1631: - Resolution: Fixed Fix Version/s: 0.8.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Thanks Patrick! JDBC driver returns wrong precision, scale, or column size for some data types -- Key: HIVE-1631 URL: https://issues.apache.org/jira/browse/HIVE-1631 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.7.0 Reporter: Steven Wong Assignee: Patrick Hunt Priority: Minor Fix For: 0.8.0 Attachments: HIVE-1631.patch For some data types, these methods return values that do not conform to the JDBC spec: org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.getPrecision(int) org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.getScale(int) org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(int) org.apache.hadoop.hive.jdbc.JdbcColumn.getColumnSize() -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2343) stats not updated for non load table desc operations
[ https://issues.apache.org/jira/browse/HIVE-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079306#comment-13079306 ] Hudson commented on HIVE-2343: -- Integrated in Hive-trunk-h0.21 #872 (See [https://builds.apache.org/job/Hive-trunk-h0.21/872/]) HIVE-2343. stats are not updated for non load table desc operations (Franklin Hu via Ning Zhang) nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1153753 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java * /hive/trunk/ql/src/test/queries/clientpositive/stats16.q * /hive/trunk/ql/src/test/results/clientpositive/stats16.q.out stats not updated for non load table desc operations -- Key: HIVE-2343 URL: https://issues.apache.org/jira/browse/HIVE-2343 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Franklin Hu Assignee: Franklin Hu Fix For: 0.8.0 Attachments: hive-2343.1.patch Bug introduced in HIVE-306 so that stats are updated only for LoadTableDesc operations. For other operations (analyze table), null ptr is thrown and stats are not updated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-trunk-h0.21 #873
See https://builds.apache.org/job/Hive-trunk-h0.21/873/changes Changes: [cws] HIVE-1631. JDBC driver returns wrong precision, scale, or column size for some data types (Patrick Hunt via cws) -- [...truncated 31562 lines...] [junit] PREHOOK: query: select count(1) as cnt from testhivedrivertable [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/jenkins/hive_2011-08-04_07-04-02_603_624209269753812736/-mr-1 [junit] Total MapReduce jobs = 1 [junit] Launching Job 1 out of 1 [junit] Number of reduce tasks determined at compile time: 1 [junit] In order to change the average load for a reducer (in bytes): [junit] set hive.exec.reducers.bytes.per.reducer=number [junit] In order to limit the maximum number of reducers: [junit] set hive.exec.reducers.max=number [junit] In order to set a constant number of reducers: [junit] set mapred.reduce.tasks=number [junit] Job running in-process (local Hadoop) [junit] Hadoop job information for null: number of mappers: 0; number of reducers: 0 [junit] 2011-08-04 07:04:05,724 null map = 100%, reduce = 100% [junit] Ended Job = job_local_0001 [junit] POSTHOOK: query: select count(1) as cnt from testhivedrivertable [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/jenkins/hive_2011-08-04_07-04-02_603_624209269753812736/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/service/tmp/hive_job_log_jenkins_201108040704_781140845.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] PREHOOK: type: LOAD [junit] PREHOOK: Output: default@testhivedrivertable [junit] Copying data from https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt [junit] Loading data to table default.testhivedrivertable [junit] POSTHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: select * from testhivedrivertable limit 10 [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/jenkins/hive_2011-08-04_07-04-07_259_1390131552402054908/-mr-1 [junit] POSTHOOK: query: select * from testhivedrivertable limit 10 [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/jenkins/hive_2011-08-04_07-04-07_259_1390131552402054908/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/service/tmp/hive_job_log_jenkins_201108040704_540965091.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit]
[jira] [Commented] (HIVE-1631) JDBC driver returns wrong precision, scale, or column size for some data types
[ https://issues.apache.org/jira/browse/HIVE-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079385#comment-13079385 ] Hudson commented on HIVE-1631: -- Integrated in Hive-trunk-h0.21 #873 (See [https://builds.apache.org/job/Hive-trunk-h0.21/873/]) HIVE-1631. JDBC driver returns wrong precision, scale, or column size for some data types (Patrick Hunt via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1153809 Files : * /hive/trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/Utils.java * /hive/trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java * /hive/trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java * /hive/trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcColumn.java * /hive/trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveResultSetMetaData.java JDBC driver returns wrong precision, scale, or column size for some data types -- Key: HIVE-1631 URL: https://issues.apache.org/jira/browse/HIVE-1631 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.7.0 Reporter: Steven Wong Assignee: Patrick Hunt Priority: Minor Fix For: 0.8.0 Attachments: HIVE-1631.patch For some data types, these methods return values that do not conform to the JDBC spec: org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.getPrecision(int) org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.getScale(int) org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(int) org.apache.hadoop.hive.jdbc.JdbcColumn.getColumnSize() -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2286) ClassCastException when building index with security.authorization turned on
[ https://issues.apache.org/jira/browse/HIVE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-2286: - Resolution: Fixed Fix Version/s: 0.8.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Syed! Hallelujah! ClassCastException when building index with security.authorization turned on Key: HIVE-2286 URL: https://issues.apache.org/jira/browse/HIVE-2286 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Fix For: 0.8.0 Attachments: HIVE-2286.1.patch, HIVE-2286.10.patch, HIVE-2286.11.patch, HIVE-2286.2.patch, HIVE-2286.6.patch, HIVE-2286.7.patch, HIVE-2286.8.patch, HIVE-2286.9.patch When trying to build an index with authorization checks turned on, hive issues the following ClassCastException: org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:540) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:848) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:293) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:385) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:392) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2319: Calling alter_table after changing partition key comment throws an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1213/ --- (Updated 2011-08-04 17:05:08.413032) Review request for hive and Paul Yang. Changes --- Removed a change in TestMetaStoreEventListener.java that somehow snuck in this diff. Summary --- Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. This addresses bug HIVE-2319. https://issues.apache.org/jira/browse/HIVE-2319 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1153927 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1153927 Diff: https://reviews.apache.org/r/1213/diff Testing --- Added some test cases to HiveMetaStore that pass. Thanks, Sohan
[jira] [Commented] (HIVE-2319) Calling alter_table after changing partition comment throws an exception
[ https://issues.apache.org/jira/browse/HIVE-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079470#comment-13079470 ] jirapos...@reviews.apache.org commented on HIVE-2319: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1213/ --- (Updated 2011-08-04 17:05:08.413032) Review request for hive and Paul Yang. Changes --- Removed a change in TestMetaStoreEventListener.java that somehow snuck in this diff. Summary --- Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. This addresses bug HIVE-2319. https://issues.apache.org/jira/browse/HIVE-2319 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1153927 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1153927 Diff: https://reviews.apache.org/r/1213/diff Testing --- Added some test cases to HiveMetaStore that pass. Thanks, Sohan Calling alter_table after changing partition comment throws an exception Key: HIVE-2319 URL: https://issues.apache.org/jira/browse/HIVE-2319 Project: Hive Issue Type: Bug Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2319.2.patch, HIVE-2319.3.patch Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2319) Calling alter_table after changing partition comment throws an exception
[ https://issues.apache.org/jira/browse/HIVE-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sohan Jain updated HIVE-2319: - Attachment: HIVE-2319.4.patch Calling alter_table after changing partition comment throws an exception Key: HIVE-2319 URL: https://issues.apache.org/jira/browse/HIVE-2319 Project: Hive Issue Type: Bug Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2319.2.patch, HIVE-2319.3.patch, HIVE-2319.4.patch Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2346) Add hooks to run when execution fails.
Add hooks to run when execution fails. -- Key: HIVE-2346 URL: https://issues.apache.org/jira/browse/HIVE-2346 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Currently, when a query fails, the Post Execution Hooks are not run. Adding hooks to be run when a query fails could allow for better logging etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2347) Make Hadoop Job ID available after task finishes executing
Make Hadoop Job ID available after task finishes executing -- Key: HIVE-2347 URL: https://issues.apache.org/jira/browse/HIVE-2347 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong After Map Reduce tasks finish the execute method (ExecDriver and BlockMergeTask) the Hadoop Job ID is inaccessible to the Driver, and hence the hooks it runs. Expose this information could help to improve logging, debugging, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2348) Group by and order by of the same columns fails semantic analysis
Group by and order by of the same columns fails semantic analysis - Key: HIVE-2348 URL: https://issues.apache.org/jira/browse/HIVE-2348 Project: Hive Issue Type: Bug Affects Versions: 0.7.1 Reporter: Edward Capriolo select offer, level_id, sum_cl_score as sumcl_floor FROM xxx GROUP BY offer, level_id, sumcl_floor ORDER BY offer, level_id, sumcl_floor limit 100; FAILED: Error in semantic analysis: Invalid Table Alias or Column Reference sumcl_floor -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2322) Add ColumnarSerDe to the list of native SerDes
[ https://issues.apache.org/jira/browse/HIVE-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079500#comment-13079500 ] Paul Yang commented on HIVE-2322: - Can you regenerate this patch? I'm getting some patch failures. Add ColumnarSerDe to the list of native SerDes -- Key: HIVE-2322 URL: https://issues.apache.org/jira/browse/HIVE-2322 Project: Hive Issue Type: Bug Components: Metastore, Serializers/Deserializers Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2322.1.patch, HIVE-2322.2.patch We store metadata about ColumnarSerDes in the metastore, so it should be considered a native SerDe. Then, column information can be retrieved from the metastore instead of from deserialization. Currently, for non-native SerDes, column comments are only shown as from deserializer. Adding ColumnarSerDe to the list of native SerDes will persist column comments. See HIVE-2171 for persisting the column comments of custom SerDes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2319) Calling alter_table after changing partition comment throws an exception
[ https://issues.apache.org/jira/browse/HIVE-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079505#comment-13079505 ] Paul Yang commented on HIVE-2319: - +1 Will test and commit Calling alter_table after changing partition comment throws an exception Key: HIVE-2319 URL: https://issues.apache.org/jira/browse/HIVE-2319 Project: Hive Issue Type: Bug Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2319.2.patch, HIVE-2319.3.patch, HIVE-2319.4.patch Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2346: Allow hooks to be run when a job fails.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ --- Review request for hive and Ning Zhang. Summary --- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346. https://issues.apache.org/jira/browse/HIVE-2346 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1153966 Diff: https://reviews.apache.org/r/1295/diff Testing --- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin
[jira] [Commented] (HIVE-2346) Add hooks to run when execution fails.
[ https://issues.apache.org/jira/browse/HIVE-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079534#comment-13079534 ] jirapos...@reviews.apache.org commented on HIVE-2346: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ --- Review request for hive and Ning Zhang. Summary --- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346. https://issues.apache.org/jira/browse/HIVE-2346 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1153966 Diff: https://reviews.apache.org/r/1295/diff Testing --- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin Add hooks to run when execution fails. -- Key: HIVE-2346 URL: https://issues.apache.org/jira/browse/HIVE-2346 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2346.1.patch.txt Currently, when a query fails, the Post Execution Hooks are not run. Adding hooks to be run when a query fails could allow for better logging etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2346) Add hooks to run when execution fails.
[ https://issues.apache.org/jira/browse/HIVE-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2346: Attachment: HIVE-2346.1.patch.txt Add hooks to run when execution fails. -- Key: HIVE-2346 URL: https://issues.apache.org/jira/browse/HIVE-2346 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2346.1.patch.txt Currently, when a query fails, the Post Execution Hooks are not run. Adding hooks to be run when a query fails could allow for better logging etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2322) Add ColumnarSerDe to the list of native SerDes
[ https://issues.apache.org/jira/browse/HIVE-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079537#comment-13079537 ] Sohan Jain commented on HIVE-2322: -- Yes, looks like some of the output.q files were updated and now conflicting. I've been re-running the test suite and re-generating them. Add ColumnarSerDe to the list of native SerDes -- Key: HIVE-2322 URL: https://issues.apache.org/jira/browse/HIVE-2322 Project: Hive Issue Type: Bug Components: Metastore, Serializers/Deserializers Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2322.1.patch, HIVE-2322.2.patch We store metadata about ColumnarSerDes in the metastore, so it should be considered a native SerDe. Then, column information can be retrieved from the metastore instead of from deserialization. Currently, for non-native SerDes, column comments are only shown as from deserializer. Adding ColumnarSerDe to the list of native SerDes will persist column comments. See HIVE-2171 for persisting the column comments of custom SerDes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2347: Added Hadoop Job ID to Task class.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1296/ --- Review request for hive and Ning Zhang. Summary --- I added a field for the Hadoop Job ID to the Task class. This will make it accessible to the Driver and hence to the hooks for logging/debugging purposes. By including it in the Task, we only need to check that the type of the task is MAPRED, before getting the job ID. I considered adding it to several places: as separate fields in ExecDriver and BlockMergeTask: this would require duplicating code, require conditions to determine the type of a task and casting to either ExecDriver or BlockMergeTask in order to get the JobID from them in the MapRedWork: this would require modifying a field in MapRedWork in the execute function, and I could not find a precedent for this This addresses bug HIVE-2347. https://issues.apache.org/jira/browse/HIVE-2347 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1153966 Diff: https://reviews.apache.org/r/1296/diff Testing --- Ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. Also, created a sample post exec hook which simply logged the JobID for every map reduce task, and verified it. Thanks, Kevin
[jira] [Commented] (HIVE-2347) Make Hadoop Job ID available after task finishes executing
[ https://issues.apache.org/jira/browse/HIVE-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079548#comment-13079548 ] Kevin Wilfong commented on HIVE-2347: - https://reviews.apache.org/r/1296/ Make Hadoop Job ID available after task finishes executing -- Key: HIVE-2347 URL: https://issues.apache.org/jira/browse/HIVE-2347 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong After Map Reduce tasks finish the execute method (ExecDriver and BlockMergeTask) the Hadoop Job ID is inaccessible to the Driver, and hence the hooks it runs. Expose this information could help to improve logging, debugging, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2347) Make Hadoop Job ID available after task finishes executing
[ https://issues.apache.org/jira/browse/HIVE-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079549#comment-13079549 ] jirapos...@reviews.apache.org commented on HIVE-2347: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1296/ --- Review request for hive and Ning Zhang. Summary --- I added a field for the Hadoop Job ID to the Task class. This will make it accessible to the Driver and hence to the hooks for logging/debugging purposes. By including it in the Task, we only need to check that the type of the task is MAPRED, before getting the job ID. I considered adding it to several places: as separate fields in ExecDriver and BlockMergeTask: this would require duplicating code, require conditions to determine the type of a task and casting to either ExecDriver or BlockMergeTask in order to get the JobID from them in the MapRedWork: this would require modifying a field in MapRedWork in the execute function, and I could not find a precedent for this This addresses bug HIVE-2347. https://issues.apache.org/jira/browse/HIVE-2347 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1153966 Diff: https://reviews.apache.org/r/1296/diff Testing --- Ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. Also, created a sample post exec hook which simply logged the JobID for every map reduce task, and verified it. Thanks, Kevin Make Hadoop Job ID available after task finishes executing -- Key: HIVE-2347 URL: https://issues.apache.org/jira/browse/HIVE-2347 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong After Map Reduce tasks finish the execute method (ExecDriver and BlockMergeTask) the Hadoop Job ID is inaccessible to the Driver, and hence the hooks it runs. Expose this information could help to improve logging, debugging, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2347) Make Hadoop Job ID available after task finishes executing
[ https://issues.apache.org/jira/browse/HIVE-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2347: Attachment: HIVE-2347.1.patch.txt Make Hadoop Job ID available after task finishes executing -- Key: HIVE-2347 URL: https://issues.apache.org/jira/browse/HIVE-2347 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2347.1.patch.txt After Map Reduce tasks finish the execute method (ExecDriver and BlockMergeTask) the Hadoop Job ID is inaccessible to the Driver, and hence the hooks it runs. Expose this information could help to improve logging, debugging, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-1989: recognize transitivity of predicates on join keys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-08-04 20:05:38.827967) Review request for hive. Changes --- Support for transitivity over n-way joins with appropriate pushing based on join type Summary --- https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1152395 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 1152395 Diff: https://reviews.apache.org/r/1228/diff Testing --- Thanks, Charles
[jira] [Updated] (HIVE-2138) Exception when no splits returned from index
[
https://issues.apache.org/jira/browse/HIVE-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi updated HIVE-2138:
-
Status: Open (was: Patch Available)
index_auto_test_if_used.q should not be in this patch.
Exception when no splits returned from index
Key: HIVE-2138
URL: https://issues.apache.org/jira/browse/HIVE-2138
Project: Hive
Issue Type: Bug
Components: Indexing
Affects Versions: 0.8.0
Reporter: Russell Melick
Assignee: Syed S. Albiz
Attachments: HIVE-2138.1.patch, HIVE-2138.2.patch, HIVE-2138.3.patch,
index_auto_test_if_used.q
Running a query that uses indexing but doesn't return any results give an
exception.
{code} java.lang.IllegalArgumentException: Can not create a Path from an
empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.init(Path.java:90)
at org.apache.hadoop.util.StringUtils.stringToPath(StringUtils.java:224)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:282)
at
org.apache.hadoop.hive.ql.index.HiveIndexedInputFormat.getSplits(HiveIndexedInputFormat.java:123)
{code}
This could potentially be fixed by creating a new empty file to use for the
splits.
Once this is fixed, the index_auto_test_if_used.q can be used.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2318) Support multiple file systems
[ https://issues.apache.org/jira/browse/HIVE-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079604#comment-13079604 ] Vaibhav Aggarwal commented on HIVE-2318: @Carl You would notice that 70% of the code deals with 1. Supporting reading with one file system and writing to another in the same query. 2. Writing directly to result directory if the file system does not support move. S3FileSystem serves as a specific example in this case because of which I choose this title. Support multiple file systems - Key: HIVE-2318 URL: https://issues.apache.org/jira/browse/HIVE-2318 Project: Hive Issue Type: New Feature Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2318.patch Currently some of the Hive tasks like MoveTask, ConditionalMergeResolver assume that the data is being copied or moved on the same file system. These operators file if the source table is in one filesystem (like HDFS) and destination table is in another file system (like s3). This patch aims at: 1. Support moving data between different file systems. 2. Add support for file systems which do not support 'move' operation like s3. 3. Remove redundant operations like moving data from and to the same location. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2318) Support multiple file systems
[ https://issues.apache.org/jira/browse/HIVE-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079606#comment-13079606 ] Vaibhav Aggarwal commented on HIVE-2318: I am thinking of writing some unit tests testing individual methods in order to simplify testing. What do you think? Support multiple file systems - Key: HIVE-2318 URL: https://issues.apache.org/jira/browse/HIVE-2318 Project: Hive Issue Type: New Feature Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2318.patch Currently some of the Hive tasks like MoveTask, ConditionalMergeResolver assume that the data is being copied or moved on the same file system. These operators file if the source table is in one filesystem (like HDFS) and destination table is in another file system (like s3). This patch aims at: 1. Support moving data between different file systems. 2. Add support for file systems which do not support 'move' operation like s3. 3. Remove redundant operations like moving data from and to the same location. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2138: Exception when no splits returned from index
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1255/ --- (Updated 2011-08-04 21:56:31.633797) Review request for hive, John Sichi and Yongqiang He. Changes --- removed stale testcase Summary --- return an empty array of input splits when we get back no results from the hiveIndexResult This addresses bug HIVE-2138. https://issues.apache.org/jira/browse/HIVE-2138 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 1a1ecd7 ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1255/diff Testing --- added index_auto_test_if_used.q to the test suite Thanks, Syed
[jira] [Commented] (HIVE-2138) Exception when no splits returned from index
[
https://issues.apache.org/jira/browse/HIVE-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079630#comment-13079630
]
jirapos...@reviews.apache.org commented on HIVE-2138:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1255/
---
(Updated 2011-08-04 21:56:31.633797)
Review request for hive, John Sichi and Yongqiang He.
Changes
---
removed stale testcase
Summary
---
return an empty array of input splits when we get back no results from the
hiveIndexResult
This addresses bug HIVE-2138.
https://issues.apache.org/jira/browse/HIVE-2138
Diffs (updated)
-
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java
1a1ecd7
ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1255/diff
Testing
---
added index_auto_test_if_used.q to the test suite
Thanks,
Syed
Exception when no splits returned from index
Key: HIVE-2138
URL: https://issues.apache.org/jira/browse/HIVE-2138
Project: Hive
Issue Type: Bug
Components: Indexing
Affects Versions: 0.8.0
Reporter: Russell Melick
Assignee: Syed S. Albiz
Attachments: HIVE-2138.1.patch, HIVE-2138.2.patch, HIVE-2138.3.patch,
HIVE-2138.4.patch, index_auto_test_if_used.q
Running a query that uses indexing but doesn't return any results give an
exception.
{code} java.lang.IllegalArgumentException: Can not create a Path from an
empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.init(Path.java:90)
at org.apache.hadoop.util.StringUtils.stringToPath(StringUtils.java:224)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:282)
at
org.apache.hadoop.hive.ql.index.HiveIndexedInputFormat.getSplits(HiveIndexedInputFormat.java:123)
{code}
This could potentially be fixed by creating a new empty file to use for the
splits.
Once this is fixed, the index_auto_test_if_used.q can be used.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2334) DESCRIBE TABLE causes NPE when hive.cli.print.header=true
[ https://issues.apache.org/jira/browse/HIVE-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reassigned HIVE-2334: - Assignee: Jakob Homan DESCRIBE TABLE causes NPE when hive.cli.print.header=true - Key: HIVE-2334 URL: https://issues.apache.org/jira/browse/HIVE-2334 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.1 Reporter: Carl Steinbach Assignee: Jakob Homan -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Support archiving for multiple partitions if the table is partitioned by multiple columns
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1259/ --- (Updated 2011-08-04 22:27:36.104846) Review request for hive, Paul Yang and namit jain. Changes --- Archived partitions counter - contains archived partitions flag. Summary --- Allowing archiving at chosen level. When table is partitioned by ds, hr, min it allows archiving at ds level, hr level and min level. Corresponding syntaxes are: ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08'); ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08', hr='11'); ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08', hr='11', min='30'); You cannot do much to archived partitions. You can read them. You cannot write to them / overwrite them. You can drop single archived partitions, but not parts of bigger archives. Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153271 trunk/data/conf/hive-site.xml 1153271 trunk/metastore/if/hive_metastore.thrift 1153271 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1153271 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1153271 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1153271 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1153271 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1153271 trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1153271 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MetaUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1153271 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1153271 trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION trunk/ql/src/test/results/clientnegative/archive1.q.out 1153271 trunk/ql/src/test/results/clientnegative/archive2.q.out 1153271 trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION
[jira] [Updated] (HIVE-2156) Improve error messages emitted during task execution
[ https://issues.apache.org/jira/browse/HIVE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed S. Albiz updated HIVE-2156: Attachment: HIVE-2156.10.patch regenerated against latest revision so fix conflicts Improve error messages emitted during task execution Key: HIVE-2156 URL: https://issues.apache.org/jira/browse/HIVE-2156 Project: Hive Issue Type: Improvement Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2156.1.patch, HIVE-2156.10.patch, HIVE-2156.2.patch, HIVE-2156.4.patch, HIVE-2156.8.patch, HIVE-2156.9.patch Follow-up to HIVE-1731 A number of issues were related to reporting errors from task execution and surfacing these in a more useful form. Currently a cryptic message with Execution Error and a return code and class name of the task is emitted. The most useful log messages here are emitted to the local logs, which can be found through jobtracker. Having either a pointer to these logs as part of the error message or the actual content would improve the usefulness substantially. It may also warrant looking into how the underlying error reporting through Hadoop is done and if more information can be propagated up from there. Specific issues raised in HIVE-1731: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask * issue was in regexp_extract syntax FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask * tried: desc table_does_not_exist; -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2110) Hive Client is indefenitely waiting for reading from Socket
[
https://issues.apache.org/jira/browse/HIVE-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasad Mujumdar reassigned HIVE-2110:
-
Assignee: Prasad Mujumdar
Hive Client is indefenitely waiting for reading from Socket
---
Key: HIVE-2110
URL: https://issues.apache.org/jira/browse/HIVE-2110
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.5.0
Environment: Hadoop 0.20.1, Hive0.5.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
Reporter: Chinna Rao Lalam
Assignee: Prasad Mujumdar
Hive Client is indefenitely waiting for reading from Socket. Thread dump i
added below.
Cause is:
In the HiveClient, when client socket is created, the read timeout is
mentioned is 0. So the socket will indefinetly wait when the machine where
Hive Server is running is shutdown or network is unplugged. The same may
not happen if the HiveServer alone is killed or gracefully shutdown. At this
time, client will get connection reset exception.
Code in HiveConnection
---
{noformat}
transport = new TSocket(host, port);
TProtocol protocol = new TBinaryProtocol(transport);
client = new HiveClient(protocol);
{noformat}
In the Client side, they send the query and wait for the response
send_execute(query,id); recv_execute(); // place where client waiting is
initiated
Thread dump:
{noformat}
main prio=10 tid=0x40111000 nid=0x3641 runnable [0x7f0d73f29000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
locked 0x7f0d5d3f0828 (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:314)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:262)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192)
at
org.apache.hadoop.hive.service.ThriftHive$Client.recv_execute(ThriftHive.java:130)
at
org.apache.hadoop.hive.service.ThriftHive$Client.execute(ThriftHive.java:109)
locked 0x7f0d5d3f0878 (a org.apache.thrift.transport.TSocket)
at
org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:218)
at
org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:154)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2318) Support multiple file systems
[ https://issues.apache.org/jira/browse/HIVE-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079694#comment-13079694 ] John Sichi commented on HIVE-2318: -- Besides mock-based unit tests, it would also be useful to have a way to configure test runs against a real S3 filesystem. (By default that configuration would be disabled for running Hive unit tests.) That way, if we want to say that a particular Hive release has working S3 support, we would make that one of the release criteria and perform the system testing on release candidates. Support multiple file systems - Key: HIVE-2318 URL: https://issues.apache.org/jira/browse/HIVE-2318 Project: Hive Issue Type: New Feature Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2318.patch Currently some of the Hive tasks like MoveTask, ConditionalMergeResolver assume that the data is being copied or moved on the same file system. These operators file if the source table is in one filesystem (like HDFS) and destination table is in another file system (like s3). This patch aims at: 1. Support moving data between different file systems. 2. Add support for file systems which do not support 'move' operation like s3. 3. Remove redundant operations like moving data from and to the same location. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2110 : Use JDBC Login timeout when connecting to Hive server.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1299/ --- Review request for hive and Carl Steinbach. Summary --- Pass the Login timeout to thrift/socket layer when connecting to the hive server. This addresses bug HIVE-2110. https://issues.apache.org/jira/browse/HIVE-2110 Diffs - jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 7c5df83 jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java c61425f Diff: https://reviews.apache.org/r/1299/diff Testing --- simple repro to verify the timeout. JDBC tests. Thanks, Prasad
[jira] [Commented] (HIVE-2110) Hive Client is indefenitely waiting for reading from Socket
[
https://issues.apache.org/jira/browse/HIVE-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079706#comment-13079706
]
jirapos...@reviews.apache.org commented on HIVE-2110:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1299/
---
Review request for hive and Carl Steinbach.
Summary
---
Pass the Login timeout to thrift/socket layer when connecting to the hive
server.
This addresses bug HIVE-2110.
https://issues.apache.org/jira/browse/HIVE-2110
Diffs
-
jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 7c5df83
jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java c61425f
Diff: https://reviews.apache.org/r/1299/diff
Testing
---
simple repro to verify the timeout. JDBC tests.
Thanks,
Prasad
Hive Client is indefenitely waiting for reading from Socket
---
Key: HIVE-2110
URL: https://issues.apache.org/jira/browse/HIVE-2110
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.5.0
Environment: Hadoop 0.20.1, Hive0.5.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
Reporter: Chinna Rao Lalam
Assignee: Prasad Mujumdar
Hive Client is indefenitely waiting for reading from Socket. Thread dump i
added below.
Cause is:
In the HiveClient, when client socket is created, the read timeout is
mentioned is 0. So the socket will indefinetly wait when the machine where
Hive Server is running is shutdown or network is unplugged. The same may
not happen if the HiveServer alone is killed or gracefully shutdown. At this
time, client will get connection reset exception.
Code in HiveConnection
---
{noformat}
transport = new TSocket(host, port);
TProtocol protocol = new TBinaryProtocol(transport);
client = new HiveClient(protocol);
{noformat}
In the Client side, they send the query and wait for the response
send_execute(query,id); recv_execute(); // place where client waiting is
initiated
Thread dump:
{noformat}
main prio=10 tid=0x40111000 nid=0x3641 runnable [0x7f0d73f29000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
locked 0x7f0d5d3f0828 (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:314)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:262)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192)
at
org.apache.hadoop.hive.service.ThriftHive$Client.recv_execute(ThriftHive.java:130)
at
org.apache.hadoop.hive.service.ThriftHive$Client.execute(ThriftHive.java:109)
locked 0x7f0d5d3f0878 (a org.apache.thrift.transport.TSocket)
at
org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:218)
at
org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:154)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1735) Extend Explode UDTF to handle Maps
[ https://issues.apache.org/jira/browse/HIVE-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1735: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Thanks Natty and Amareshwari! Extend Explode UDTF to handle Maps -- Key: HIVE-1735 URL: https://issues.apache.org/jira/browse/HIVE-1735 Project: Hive Issue Type: New Feature Components: UDF Reporter: Carl Steinbach Assignee: Jonathan Natkins Fix For: 0.8.0 Attachments: HIVE-1735.1.patch, HIVE-1735.2.patch.txt, HIVE-1735.2.patch.txt, HIVE-1735.3.patch.txt The explode() UDTF currently only accepts arrays as input. We should modify it so that it can also handle map inputs, in which case it will output two columns corresponding to the key and value fields. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-1989: recognize transitivity of predicates on join keys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-08-05 01:17:23.359430) Review request for hive. Changes --- Floyd warshall fix, unit tests fix Summary --- https://issues.apache.org/jira/browse/HIVE-1989 This addresses bug HIVE-1989. https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cluster.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join16.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/louter_join_ppr.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join1.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join2.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_6.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union22.q.out 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join1.q.xml 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join2.q.xml 1153992 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join3.q.xml 1153992 Diff: https://reviews.apache.org/r/1228/diff Testing --- Thanks, Charles
[jira] [Updated] (HIVE-2334) DESCRIBE TABLE causes NPE when hive.cli.print.header=true
[ https://issues.apache.org/jira/browse/HIVE-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HIVE-2334: -- Status: Patch Available (was: Open) DESCRIBE TABLE causes NPE when hive.cli.print.header=true - Key: HIVE-2334 URL: https://issues.apache.org/jira/browse/HIVE-2334 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.1 Reporter: Carl Steinbach Assignee: Jakob Homan Attachments: h2334.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2171) Allow custom serdes to set field comments
[
https://issues.apache.org/jira/browse/HIVE-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HIVE-2171:
--
Status: Patch Available (was: Open)
Allow custom serdes to set field comments
-
Key: HIVE-2171
URL: https://issues.apache.org/jira/browse/HIVE-2171
Project: Hive
Issue Type: Improvement
Affects Versions: 0.7.0
Reporter: Jakob Homan
Assignee: Jakob Homan
Attachments: HIVE-2171-2.patch, HIVE-2171.patch
Currently, while serde implementations can set a field's name, they can't set
its comment. These are set in the metastore utils to {{(from
deserializer)}}. For those serdes that can provide meaningful comments for a
field, they should be propagated to the table description. These
serde-provided comments could be prepended to (from deserializer) if others
feel that's a meaningful distinction. This change involves updating
{{StructField}} to support a (possibly null) comment field and then
propagating this change out to the myriad places {{StructField}} is thrown
around.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2258) Honor -S flag during hadoop rmr command
[ https://issues.apache.org/jira/browse/HIVE-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079753#comment-13079753 ] Jakob Homan commented on HIVE-2258: --- Vaibhav - can you please create a review board for this patch? Also, I'm a bit confused - won't this hide the output for the rmr regardless of -S is set? Moreover, the current Hive documentation says the -S will only emit data, but isn't the message from rmr (generally that a file has been moved to the trash) is also data? If not, should we be consistent and suppress all the output of the shell commands? Honor -S flag during hadoop rmr command --- Key: HIVE-2258 URL: https://issues.apache.org/jira/browse/HIVE-2258 Project: Hive Issue Type: Bug Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2258.patch Currently even if -S flag is specified, the output of hadoop -rmr command is printed to the screen. The reason is that the command writes output to screen instead of log file. I have fixed the problem by temporarily redirecting the output for that command. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2350) Improve RCFile Read Speed
[ https://issues.apache.org/jira/browse/HIVE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated HIVE-2350: Attachment: rcfile-2011-08-04.diff Improve RCFile Read Speed - Key: HIVE-2350 URL: https://issues.apache.org/jira/browse/HIVE-2350 Project: Hive Issue Type: Improvement Reporter: Tim Armstrong Assignee: Tim Armstrong Priority: Minor Attachments: rcfile-2011-08-04.diff Original Estimate: 0h Remaining Estimate: 0h By tweaking the RCFile$Reader implementation to allow more efficient memory access I was able to reduce CPU usage. I measured the speed required to scan a gzipped RCFile, decompress and assemble into records. CPU time was reduced by about 7% for a full table scan, An improvement of about 2% was realised when a smaller subset of columns (3-5 out of tens) were selected. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2298) Fix UDAFPercentile to tolerate null percentiles
[ https://issues.apache.org/jira/browse/HIVE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-2298: -- Resolution: Fixed Fix Version/s: 0.8.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks Vaibhav ! Fix UDAFPercentile to tolerate null percentiles --- Key: HIVE-2298 URL: https://issues.apache.org/jira/browse/HIVE-2298 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.7.0 Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Fix For: 0.8.0 Attachments: HIVE-2298-2.patch, HIVE-2298-3.patch, HIVE-2298.patch UDAFPercentile when passed null percentile list will throw a null pointer exception. Submitting a small fix for that. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2138) Exception when no splits returned from index
[
https://issues.apache.org/jira/browse/HIVE-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi updated HIVE-2138:
-
Status: Open (was: Patch Available)
index_auto_empty needs its log updated:
[junit] PREHOOK: type: ALTERINDEX_REBUILD
[junit] ---
[junit] PREHOOK: type: QUERY
[junit] 23c23
[junit] POSTHOOK: type: ALTERINDEX_REBUILD
[junit] ---
Exception when no splits returned from index
Key: HIVE-2138
URL: https://issues.apache.org/jira/browse/HIVE-2138
Project: Hive
Issue Type: Bug
Components: Indexing
Affects Versions: 0.8.0
Reporter: Russell Melick
Assignee: Syed S. Albiz
Attachments: HIVE-2138.1.patch, HIVE-2138.2.patch, HIVE-2138.3.patch,
HIVE-2138.4.patch, index_auto_test_if_used.q
Running a query that uses indexing but doesn't return any results give an
exception.
{code} java.lang.IllegalArgumentException: Can not create a Path from an
empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.init(Path.java:90)
at org.apache.hadoop.util.StringUtils.stringToPath(StringUtils.java:224)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:282)
at
org.apache.hadoop.hive.ql.index.HiveIndexedInputFormat.getSplits(HiveIndexedInputFormat.java:123)
{code}
This could potentially be fixed by creating a new empty file to use for the
splits.
Once this is fixed, the index_auto_test_if_used.q can be used.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
