[jira] Updated: (HIVE-1508) Add cleanup method to HiveHistory class
[ https://issues.apache.org/jira/browse/HIVE-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-1508: -- Attachment: hive-1508-1-patch.txt Add cleanup method to HiveHistory class --- Key: HIVE-1508 URL: https://issues.apache.org/jira/browse/HIVE-1508 Project: Hive Issue Type: Bug Components: Metastore Reporter: Anurag Phadke Assignee: Edward Capriolo Priority: Blocker Fix For: 0.7.0 Attachments: hive-1508-1-patch.txt Running hive server for long time 90 minutes results in too many open file-handles, eventually causing the server to crash as the server runs out of file handle. Actual bug as described by Carl Steinbach: the hive_job_log_* files are created by the HiveHistory class. This class creates a PrintWriter for writing to the file, but never closes the writer. It looks like we need to add a cleanup method to HiveHistory that closes the PrintWriter and does any other necessary cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1508) Add cleanup method to HiveHistory class
[ https://issues.apache.org/jira/browse/HIVE-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-1508: -- Status: Patch Available (was: Open) Did two things here. First random can generate the same result if called twice in the same millisecond. We guard against that with a while condition. Secondly leverage a finalizer to deal with clean up. Might be a nicer way, but not sure what is the best way to go about that. No easy way to unit test here, but you could run with lsof and determine if handles are ever closing. Add cleanup method to HiveHistory class --- Key: HIVE-1508 URL: https://issues.apache.org/jira/browse/HIVE-1508 Project: Hive Issue Type: Bug Components: Metastore Reporter: Anurag Phadke Assignee: Edward Capriolo Priority: Blocker Fix For: 0.7.0 Attachments: hive-1508-1-patch.txt Running hive server for long time 90 minutes results in too many open file-handles, eventually causing the server to crash as the server runs out of file handle. Actual bug as described by Carl Steinbach: the hive_job_log_* files are created by the HiveHistory class. This class creates a PrintWriter for writing to the file, but never closes the writer. It looks like we need to add a cleanup method to HiveHistory that closes the PrintWriter and does any other necessary cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1692) FetchOperator.getInputFormatFromCache hides causal exception
[ https://issues.apache.org/jira/browse/HIVE-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966315#action_12966315 ] Edward Capriolo commented on HIVE-1692: --- +1. Will commit if tests pass. Philip, make sure to click SUBMIT_PATCH when done with an issue to signal a review FetchOperator.getInputFormatFromCache hides causal exception Key: HIVE-1692 URL: https://issues.apache.org/jira/browse/HIVE-1692 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.7.0 Reporter: Philip Zeyliger Priority: Minor Fix For: 0.7.0 Attachments: HIVE-1692.patch.txt There's a line in FetchOperator.getInputFormatFromCache that catches all exceptions and re-throws IOException instead, hiding the original cause. I ran into this, naturally, and wish to fix it. Patch below is trivial. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1824) create a new ZooKeeper instance when retrying lock, and more info for debug
create a new ZooKeeper instance when retrying lock, and more info for debug --- Key: HIVE-1824 URL: https://issues.apache.org/jira/browse/HIVE-1824 Project: Hive Issue Type: Improvement Reporter: He Yongqiang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1825) Different defaults for hive.metastore.local
Different defaults for hive.metastore.local
---
Key: HIVE-1825
URL: https://issues.apache.org/jira/browse/HIVE-1825
Project: Hive
Issue Type: Bug
Components: Configuration
Affects Versions: 0.6.0
Reporter: Lars Francke
hive-default.xml sets {{hive.metastore.local}} to {{true}}. In the code however
there is this:
{code:title=HiveMetaStoreClient.java}
boolean localMetaStore = conf.getBoolean(hive.metastore.local, false);
{code}
This leads to different behaviour depending on whether hbase-default.xml is on
the classpath or not.which can lead to some confusion ;-)
I can supply a patch - should be pretty similar. I just don't know what the
real default should be. My guess would be {{true}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Possible Bug with Conv()
Hi there, I've got a column which contains hex data in string format. I run the following query: select conv([column], 16, 10) from [table] limit 5; The output, though, seems to mess up if two or more consecutive columns contain the same data. To test this out I did: select conv(a, 16, 10) from [table] limit 5; In theory (unless I'm mistaken) this should produce five rows with 10. Instead, it produces: OK 10 794 15002 3441418 913786650 Time taken: 17.235 seconds Can you offer me any suggestions? Thanks, Annie Graebner Return Path
[jira] Updated: (HIVE-1824) create a new ZooKeeper instance when retrying lock, and more info for debug
[ https://issues.apache.org/jira/browse/HIVE-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1824: --- Attachment: hive-1824.1.patch create a new ZooKeeper instance when retrying lock, and more info for debug --- Key: HIVE-1824 URL: https://issues.apache.org/jira/browse/HIVE-1824 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Attachments: hive-1824.1.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1824) create a new ZooKeeper instance when retrying lock, and more info for debug
[ https://issues.apache.org/jira/browse/HIVE-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang reassigned HIVE-1824: -- Assignee: He Yongqiang create a new ZooKeeper instance when retrying lock, and more info for debug --- Key: HIVE-1824 URL: https://issues.apache.org/jira/browse/HIVE-1824 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: hive-1824.1.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Butler updated HIVE-1648: -- Attachment: HIVE-1648.4.patch Automatically gathering stats when reading a table/partition Key: HIVE-1648 URL: https://issues.apache.org/jira/browse/HIVE-1648 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Paul Butler Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, HIVE-1648.patch HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966354#action_12966354 ] Paul Butler commented on HIVE-1648: --- Changes made. Note that subqueries are not piggybacked, but tests are there to make sure they still run when hive.stats.autogather=true. Automatically gathering stats when reading a table/partition Key: HIVE-1648 URL: https://issues.apache.org/jira/browse/HIVE-1648 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Paul Butler Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, HIVE-1648.patch HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1819) maintain lastAccessTime in the metastore
[ https://issues.apache.org/jira/browse/HIVE-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1819: - Attachment: hive.1819.4.patch maintain lastAccessTime in the metastore Key: HIVE-1819 URL: https://issues.apache.org/jira/browse/HIVE-1819 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1819.1.patch, hive.1819.2.patch, hive.1819.3.patch, hive.1819.4.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1826) StatsTask updates the table/partition object leaving a inconsistent version in hooks
[ https://issues.apache.org/jira/browse/HIVE-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966357#action_12966357 ] Namit Jain commented on HIVE-1826: -- The inputs and outputs from the ReadEntity and WriteEntity are passed to the hooks. However, the StatsTask may have updated these objects. Isn't it possible that the hooks (post execution) will see a stale version of this data ? And, if these hooks update these objects and write them back to the metastore, the Stats changes will be lost. StatsTask updates the table/partition object leaving a inconsistent version in hooks Key: HIVE-1826 URL: https://issues.apache.org/jira/browse/HIVE-1826 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Ning Zhang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1824) create a new ZooKeeper instance when retrying lock, and more info for debug
[ https://issues.apache.org/jira/browse/HIVE-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1824: --- Attachment: hive-1824.2.patch create a new ZooKeeper instance when retrying lock, and more info for debug --- Key: HIVE-1824 URL: https://issues.apache.org/jira/browse/HIVE-1824 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: hive-1824.1.patch, hive-1824.2.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1824) create a new ZooKeeper instance when retrying lock, and more info for debug
[ https://issues.apache.org/jira/browse/HIVE-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1824: --- Attachment: (was: hive-1824.2.patch) create a new ZooKeeper instance when retrying lock, and more info for debug --- Key: HIVE-1824 URL: https://issues.apache.org/jira/browse/HIVE-1824 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: hive-1824.1.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966367#action_12966367 ] Paul Butler commented on HIVE-1763: --- I was concerned about breaking DROP TABLE's idempotence, so rather than throwing an exception I just print the error to the console. If someone can suggest a better approach I'll do it. drop table (or view) should issue warning if table doesn't exist Key: HIVE-1763 URL: https://issues.apache.org/jira/browse/HIVE-1763 Project: Hive Issue Type: Improvement Components: Metastore Reporter: dan f Priority: Minor Attachments: HIVE-1763.patch drop table reports OK even if the table doesn't exist. Better to report something like mysql's Unknown table 'foo' so that, e.g., unwanted tables (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Butler updated HIVE-1763: -- Assignee: Paul Butler drop table (or view) should issue warning if table doesn't exist Key: HIVE-1763 URL: https://issues.apache.org/jira/browse/HIVE-1763 Project: Hive Issue Type: Improvement Components: Metastore Reporter: dan f Assignee: Paul Butler Priority: Minor Attachments: HIVE-1763.patch drop table reports OK even if the table doesn't exist. Better to report something like mysql's Unknown table 'foo' so that, e.g., unwanted tables (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966391#action_12966391 ] Carl Steinbach commented on HIVE-1526: -- bq. A new problem is that if you run ant -Dthrift.home=thrift-0.5-home thriftif and then run ant package, there will be compilation error. I was unable to reproduce the compilation error. Are you sure that you're using Thrift 0.5.0? Did you apply the complete patch before executing the thriftif target? Which files were modified as a result of executing the thriftif target? bq. A similar problem can be reproduced by removing all svn deleted files under serde/src and run ant package. I don't think I understand what you did. How can you remove files that were already svn deleted? Or are you saying that you did something similar to rm -rf serde/src/gen-*? If so that's going to produce errors because in the process you also blew away protobuf generated code located in the serde/src/gen-java directory. bq. I debugged it a little bit and found it may be related to the thrift generated java and java:bean under serde. The code in serde/src/gen-java is protobuf code, not Thrift code. This directory used to contain Thrift generated code, but that code now gets placed in the gen-javabean directory. If this directory contains any Thrift code then I think the patch was not applied correctly. Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.7.0 Attachments: compile.err, HIVE-1526-complete.4.patch.txt, HIVE-1526-complete.5.patch.txt, HIVE-1526-no-codegen.3.patch.txt, HIVE-1526-no-codegen.4.patch.txt, HIVE-1526-no-codegen.5.patch.txt, HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1526) Hive should depend on a release version of Thrift
[
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ning Zhang updated HIVE-1526:
-
Attachment: test.log
Carl, I figured out why there is a thrift compilation error: the
serde/build.xml:70 need to changed from
executable=thrift to executable=${thrift.home}/bin/thrift.
I have a different version of thrift installed on /usr/local/bin, and that was
used to generate.
Even after resolving this the unit tests failed with the same error (dynamic
serde). I'm attaching the full log file here.
Hive should depend on a release version of Thrift
-
Key: HIVE-1526
URL: https://issues.apache.org/jira/browse/HIVE-1526
Project: Hive
Issue Type: Task
Components: Build Infrastructure, Clients
Reporter: Carl Steinbach
Assignee: Carl Steinbach
Fix For: 0.7.0
Attachments: compile.err, HIVE-1526-complete.4.patch.txt,
HIVE-1526-complete.5.patch.txt, HIVE-1526-no-codegen.3.patch.txt,
HIVE-1526-no-codegen.4.patch.txt, HIVE-1526-no-codegen.5.patch.txt,
HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar,
libthrift.jar, serde2_test.patch, svn_rm.sh, test.log, thrift-0.5.0.jar,
thrift-fb303-0.5.0.jar
Hive should depend on a release version of Thrift, and ideally it should use
Ivy to resolve this dependency.
The Thrift folks are working on adding Thrift artifacts to a maven repository
here: https://issues.apache.org/jira/browse/THRIFT-363
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1819) maintain lastAccessTime in the metastore
[ https://issues.apache.org/jira/browse/HIVE-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1819: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed! Thanks Namit! maintain lastAccessTime in the metastore Key: HIVE-1819 URL: https://issues.apache.org/jira/browse/HIVE-1819 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1819.1.patch, hive.1819.2.patch, hive.1819.3.patch, hive.1819.4.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut reassigned HIVE-1675: -- Assignee: Bennie Schut SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-1675 started by Bennie Schut. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1675: --- Attachment: HIVE-1675.patch We needed to close the OutputStream (or flush it) before another process can read the data. This fixed the SAXParserException we got. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1675: --- Fix Version/s: 0.7.0 Status: Patch Available (was: In Progress) SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1675: --- Status: Open (was: Patch Available) doesn't look like it actually got solved. Testing it a bit more. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1828) show locks should not use getTable()/getPartition
show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1822) Hive Conf variables should be relative to the dfs
[ https://issues.apache.org/jira/browse/HIVE-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1822: - Status: Patch Available (was: Open) Hive Conf variables should be relative to the dfs - Key: HIVE-1822 URL: https://issues.apache.org/jira/browse/HIVE-1822 Project: Hive Issue Type: Improvement Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1822.1.patch Currently, the following parameter: hive.metastore.warehouse.dir refers the path completely. It becomes difficult to maintain if a mapping from Hive Database - DFS is added. This is needed for multi data-center support from Hive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1822) Hive Conf variables should be relative to the dfs
[ https://issues.apache.org/jira/browse/HIVE-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1822: - Attachment: hive.1822.1.patch Hive Conf variables should be relative to the dfs - Key: HIVE-1822 URL: https://issues.apache.org/jira/browse/HIVE-1822 Project: Hive Issue Type: Improvement Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1822.1.patch Currently, the following parameter: hive.metastore.warehouse.dir refers the path completely. It becomes difficult to maintain if a mapping from Hive Database - DFS is added. This is needed for multi data-center support from Hive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1290#action_1290 ] Namit Jain commented on HIVE-1648: -- I dont see any new tests Automatically gathering stats when reading a table/partition Key: HIVE-1648 URL: https://issues.apache.org/jira/browse/HIVE-1648 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Paul Butler Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, HIVE-1648.patch HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1829) TestRemoteMetaStore fails if machine has multiple IPs
TestRemoteMetaStore fails if machine has multiple IPs
-
Key: HIVE-1829
URL: https://issues.apache.org/jira/browse/HIVE-1829
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.6.0
Reporter: Edward Capriolo
Notice how Running metastore! appears twice.
{noformat}
test:
[junit] Running org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore
[junit] BR.recoverFromMismatchedToken
[junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 36.697 sec
[junit] Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
[junit] Running metastore!
[junit] Running metastore!
[junit] org.apache.thrift.transport.TTransportException: Could not create
ServerSocket on address 0.0.0.0/0.0.0.0:29083.
[junit] at
org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:98)
[junit] at
org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:79)
[junit] at
org.apache.hadoop.hive.metastore.TServerSocketKeepAlive.init(TServerSocketKeepAlive.java:34)
[junit] at
org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:2189)
[junit] at
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore$RunMS.run(TestRemoteHiveMetaStore.java:35)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit] Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Test org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
FAILED (crashed)
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1822) Hive Conf variables should be relative to the dfs
[
https://issues.apache.org/jira/browse/HIVE-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966702#action_12966702
]
Ning Zhang commented on HIVE-1822:
--
In build-common.xml:401, why ${test.warehouse.scheme} is removed? Isn't it used
by pfile?
Hive Conf variables should be relative to the dfs
-
Key: HIVE-1822
URL: https://issues.apache.org/jira/browse/HIVE-1822
Project: Hive
Issue Type: Improvement
Reporter: Namit Jain
Assignee: Namit Jain
Attachments: hive.1822.1.patch
Currently, the following parameter:
hive.metastore.warehouse.dir
refers the path completely.
It becomes difficult to maintain if a mapping from Hive Database - DFS is
added.
This is needed for multi data-center support from Hive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1828) show locks should not use getTable()/getPartition
[ https://issues.apache.org/jira/browse/HIVE-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1828: --- Status: Patch Available (was: Open) show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang Attachments: HIVE-1828.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1828) show locks should not use getTable()/getPartition
[ https://issues.apache.org/jira/browse/HIVE-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966847#action_12966847 ] Namit Jain commented on HIVE-1828: -- One minor comment: In case of show locks T extended; Does anyone check that the table exists ? The DDLTask can do that before calling zookeeper show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang Attachments: HIVE-1828.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1828) show locks should not use getTable()/getPartition
[ https://issues.apache.org/jira/browse/HIVE-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966872#action_12966872 ] He Yongqiang commented on HIVE-1828: For show locks T extended;, DDLTask is now validating the table and partition information in getHiveLockObject(). show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang Attachments: HIVE-1828.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1828) show locks should not use getTable()/getPartition
[ https://issues.apache.org/jira/browse/HIVE-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12966981#action_12966981 ] Namit Jain commented on HIVE-1828: -- can you add the new patch ? also, can you add a negative test (if you have not done so already) ? show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang Attachments: HIVE-1828.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1828) show locks should not use getTable()/getPartition
[ https://issues.apache.org/jira/browse/HIVE-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1828: - Status: Open (was: Patch Available) show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang Attachments: HIVE-1828.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1830) mappers in group followed by joins may die OOM
mappers in group followed by joins may die OOM -- Key: HIVE-1830 URL: https://issues.apache.org/jira/browse/HIVE-1830 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Liyin Tang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1830) mappers in group followed by joins may die OOM
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12967105#action_12967105 ] Namit Jain commented on HIVE-1830: -- After HIVE-1642, joins are automatically converted into map-joins at physical optimization time. However, this may lead to problems. For eg: consider the query: select T1.val, count(1) from T1 join T2 on T1.key=T2.key group by T1.val This will have 2 map-reduce jobs, one for the join and the other for group by. Before HIVE-1642, the partial group for aggregation will be performed in the reducer where the join is performed. However, after HIVE-1642, the same will be performed in the mapper. The local task will confirm that there is just enough memory to hold the map-join data. Hoever, it does not take into account the memory needed for partial group by. So, in case there is group by followed by join, it is a good idea to reduce the memory given to the local task to validate if there is enough memory to fit small table - it can be controlled by a new configuration paramter, but it can be some default: say 70% of total memory (instead of 90%). Also, the group by may still run out of memory, so it might be a good idea to check in group by for free memory and periodically flush memory mappers in group followed by joins may die OOM -- Key: HIVE-1830 URL: https://issues.apache.org/jira/browse/HIVE-1830 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Liyin Tang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1831) Add a option to run task to check map-join possibility in non-local mode
Add a option to run task to check map-join possibility in non-local mode Key: HIVE-1831 URL: https://issues.apache.org/jira/browse/HIVE-1831 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Liyin Tang In HIVE-1642, we run a local task to figure out if the small table can be held in memory, and then convert the join into a map-join. However, this can be a good idea for thin clients (which may not have enough memory). This should be made configurable - where the default can still be to run the task locally on the client machine, but an option should be added for thin clients, where the task would be run as a map-only task -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12967142#action_12967142 ] Bennie Schut commented on HIVE-1675: The good news is I'm unable to reproduce this once I set hive.exec.parallel=false. Perhaps it's cleaning uo some things during one of the steps while the other steps still need it? hive.exec.parallel works great for non local stuff for us but the combination of local and parallel seems like it has some issues. I was getting all kinds of different exceptions on local mode like the saxparser but also: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_local_0001/attempt_local_0001_m_00_0/output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:50) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:193) and: java.io.FileNotFoundException: HIVE_PLAN1fe0a487-2e46-44e4-a557-9b3c9dddac07 (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:106) at java.io.FileInputStream.init(FileInputStream.java:66) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:155) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:238) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:244) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:139) I only get these issues when enabling parallel and local. At least an angle to look at. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1526: - Attachment: HIVE-1526-no-codegen.6.patch.txt HIVE-1526-complete.6.patch.txt Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.7.0 Attachments: compile.err, HIVE-1526-complete.4.patch.txt, HIVE-1526-complete.5.patch.txt, HIVE-1526-complete.6.patch.txt, HIVE-1526-no-codegen.3.patch.txt, HIVE-1526-no-codegen.4.patch.txt, HIVE-1526-no-codegen.5.patch.txt, HIVE-1526-no-codegen.6.patch.txt, HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, serde2_test.patch, svn_rm.sh, test.log, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12967147#action_12967147 ] Carl Steinbach commented on HIVE-1526: -- @Ning: The errors in join_filters.q and bucketmapjoin6.q are due to the change in SMBMapJoinOperator.getType(), which I have reverted in the latest patch. I am currently running tests. Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.7.0 Attachments: compile.err, HIVE-1526-complete.4.patch.txt, HIVE-1526-complete.5.patch.txt, HIVE-1526-complete.6.patch.txt, HIVE-1526-no-codegen.3.patch.txt, HIVE-1526-no-codegen.4.patch.txt, HIVE-1526-no-codegen.5.patch.txt, HIVE-1526-no-codegen.6.patch.txt, HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, serde2_test.patch, svn_rm.sh, test.log, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1832) Dynamically allocate and measure memory usage when a map join op followed by a group by op
Dynamically allocate and measure memory usage when a map join op followed by a group by op -- Key: HIVE-1832 URL: https://issues.apache.org/jira/browse/HIVE-1832 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Liyin Tang Assignee: Liyin Tang Right now, if a map join operator followed by a map-side group by, this map reduce task will be memory intensive task. Memory usage should be carefully measured and bounded in order not to run out of memory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1832) Dynamically allocate and measure memory usage when a map join op followed by a group by op
[ https://issues.apache.org/jira/browse/HIVE-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12967268#action_12967268 ] Liyin Tang commented on HIVE-1832: -- Duplicate of Hive-1830 Dynamically allocate and measure memory usage when a map join op followed by a group by op -- Key: HIVE-1832 URL: https://issues.apache.org/jira/browse/HIVE-1832 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Liyin Tang Assignee: Liyin Tang Right now, if a map join operator followed by a map-side group by, this map reduce task will be memory intensive task. Memory usage should be carefully measured and bounded in order not to run out of memory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1827) Audit how many queries will be run in the local mode
[ https://issues.apache.org/jira/browse/HIVE-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Tang updated HIVE-1827: - Attachment: hive-1827-1.patch Add a new attribute isLocalMode in Task. Audit how many queries will be run in the local mode Key: HIVE-1827 URL: https://issues.apache.org/jira/browse/HIVE-1827 Project: Hive Issue Type: New Feature Reporter: Liyin Tang Assignee: Liyin Tang Attachments: hive-1827-1.patch Hive can run query in local mode. It would be nice to track and audit how many queries will be run in the local mode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1827) Audit how many queries will be run in the local mode
[ https://issues.apache.org/jira/browse/HIVE-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968359#action_12968359 ] He Yongqiang commented on HIVE-1827: +1. Audit how many queries will be run in the local mode Key: HIVE-1827 URL: https://issues.apache.org/jira/browse/HIVE-1827 Project: Hive Issue Type: New Feature Reporter: Liyin Tang Assignee: Liyin Tang Attachments: hive-1827-1.patch Hive can run query in local mode. It would be nice to track and audit how many queries will be run in the local mode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1833) Task-cleanup task should be disabled
Task-cleanup task should be disabled Key: HIVE-1833 URL: https://issues.apache.org/jira/browse/HIVE-1833 Project: Hive Issue Type: Improvement Components: Server Infrastructure Reporter: Scott Chen Assignee: Scott Chen Currently when task fails, a cleanup attempt will be scheduled right after that. This is unnecessary and increase the latency. MapReduce will allow disabling this (see MAPREDUCE-2206). After that patch is committed, we should set the JobConf in HIVE to disable cleanup task. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968369#action_12968369 ] Ning Zhang commented on HIVE-1526: -- Hi Carl, the latest patch has a compilation error building odbc (ant -Dthrift.home=... compile-cpp). I think it is due to the fact that the thrift generated code are under src/gen/thrift/gen-cpp etc. You may need to change the Makefile to reflect that. Also there are still exceptions on dynamic serde. When you run 'ant -Dthrift.home=... -Dtestcase=TestDynamicSerde' it throws org.apache.thrift.transport.TTransportException (they also appear in the test.log I uploaded before) although at last ant showed 'BUILD SUCCESSFUL'. In the trunk there is no such exception. I agree that dynamic serde itself may not as important and we are considering to deprecate it, but it is one of the few (if not the only) test cases in our unit tests that uses the thrift library. I think we should find out the reason why there are exceptions before we can commit the changes. Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.7.0 Attachments: compile.err, HIVE-1526-complete.4.patch.txt, HIVE-1526-complete.5.patch.txt, HIVE-1526-complete.6.patch.txt, HIVE-1526-no-codegen.3.patch.txt, HIVE-1526-no-codegen.4.patch.txt, HIVE-1526-no-codegen.5.patch.txt, HIVE-1526-no-codegen.6.patch.txt, HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, serde2_test.patch, svn_rm.sh, test.log, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1488) CombineHiveInputFormat for hadoop-19 is broken
[
https://issues.apache.org/jira/browse/HIVE-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ning Zhang resolved HIVE-1488.
--
Resolution: Won't Fix
Hive 0.7 now only support hadoop 0.20.0+. So no need to fix this JIRA on trunk
anymore.
CombineHiveInputFormat for hadoop-19 is broken
--
Key: HIVE-1488
URL: https://issues.apache.org/jira/browse/HIVE-1488
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Joydeep Sen Sarma
Assignee: Ning Zhang
I don't if anyone is using it. After making some recent testing related
changes in HIVE-1408, combine[12].q are no longer working when testing
against 19. I have seen them fail earlier as well and not investigated.
Looking at the code, it seems pretty hokey:
getInputPathsShim():
Path[] newPaths = new Path[paths.length];
// remove file:
for (int pos = 0; pos paths.length; pos++) {
newPaths[pos] = new Path(paths[pos].toString().substring(5));
}
since we are no longer using 'file:' namespace for test warehouse, this is
broke. But this would be broken against any hdfs instance it would seem(?).
Also not clear what we are trying to do here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1828) show locks should not use getTable()/getPartition
[ https://issues.apache.org/jira/browse/HIVE-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1828: --- Status: Patch Available (was: Open) show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang Attachments: HIVE-1828.2.patch, HIVE-1828.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1828) show locks should not use getTable()/getPartition
[ https://issues.apache.org/jira/browse/HIVE-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1828: --- Attachment: HIVE-1828.2.patch show locks should not use getTable()/getPartition -- Key: HIVE-1828 URL: https://issues.apache.org/jira/browse/HIVE-1828 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang Attachments: HIVE-1828.2.patch, HIVE-1828.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Hudson build is back to normal : Hive-trunk-h0.20 #434
See https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/434/
[jira] Created: (HIVE-1834) more debugging for locking
more debugging for locking -- Key: HIVE-1834 URL: https://issues.apache.org/jira/browse/HIVE-1834 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Along with the time and the queryid, it might be a good idea to log if the lock was acquired explicitly (by a lock command) or implicitly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1823) upgrade the database thrift interface to allow parameters key-value pairs
[ https://issues.apache.org/jira/browse/HIVE-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968463#action_12968463 ] Namit Jain commented on HIVE-1823: -- +1 running tests upgrade the database thrift interface to allow parameters key-value pairs - Key: HIVE-1823 URL: https://issues.apache.org/jira/browse/HIVE-1823 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-1823.patch In order to store data center specify parameters to Hive database, it is desirable to extend Hive database thrift interface with a parameters map similar to Table and Partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1835) Better auto-complete for Hive
Better auto-complete for Hive - Key: HIVE-1835 URL: https://issues.apache.org/jira/browse/HIVE-1835 Project: Hive Issue Type: New Feature Components: CLI Reporter: Paul Butler Assignee: Paul Butler Priority: Minor - Add functions and keywords to auto-complete list - Make Hive auto-complete aware of Hive delimiters (eg. whitespace, parentheses) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1835) Better auto-complete for Hive
[ https://issues.apache.org/jira/browse/HIVE-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Butler updated HIVE-1835: -- Attachment: HIVE-1835.patch Better auto-complete for Hive - Key: HIVE-1835 URL: https://issues.apache.org/jira/browse/HIVE-1835 Project: Hive Issue Type: New Feature Components: CLI Reporter: Paul Butler Assignee: Paul Butler Priority: Minor Attachments: HIVE-1835.patch - Add functions and keywords to auto-complete list - Make Hive auto-complete aware of Hive delimiters (eg. whitespace, parentheses) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1830) mappers in group followed by joins may die OOM
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Tang updated HIVE-1830: - Attachment: hive-1830-2.patch Add a new test: auto_join26.q mappers in group followed by joins may die OOM -- Key: HIVE-1830 URL: https://issues.apache.org/jira/browse/HIVE-1830 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Liyin Tang Attachments: hive-1830-1.patch, hive-1830-2.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1823) upgrade the database thrift interface to allow parameters key-value pairs
[ https://issues.apache.org/jira/browse/HIVE-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang updated HIVE-1823: - Attachment: HIVE-1823.2.patch Fixed a simple bug in the previous patch: the new DATABSE_PARAMS table should use both DB_ID and KEY_COLUMN as the primary key. Since the index key max length is 767 in mysql, the key column's length is reduced to 180 from 256 in case of the db character set is utf-8. upgrade the database thrift interface to allow parameters key-value pairs - Key: HIVE-1823 URL: https://issues.apache.org/jira/browse/HIVE-1823 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-1823.2.patch, HIVE-1823.patch In order to store data center specify parameters to Hive database, it is desirable to extend Hive database thrift interface with a parameters map similar to Table and Partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1079) CREATE VIEW followup: derive dependencies on underlying base table partitions from view definition
[ https://issues.apache.org/jira/browse/HIVE-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1079: Assignee: John Sichi (was: Carl Steinbach) CREATE VIEW followup: derive dependencies on underlying base table partitions from view definition --- Key: HIVE-1079 URL: https://issues.apache.org/jira/browse/HIVE-1079 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi When querying a view, it would be useful to know which underlying base table partitions it depends on in order to know how fresh the result is (or to be able to wait until all of those partitions have been loaded consistently). The task is to come up with a way to perform this analysis automatically (possibly overconservatively), or alternately to let the view creator annotate the view definition with this dependency information, or some combination of the two. Note that this would be useful for any complex query which directly accesses base tables (not just view definitions). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1830) mappers in group followed by joins may die OOM
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Tang updated HIVE-1830: - Attachment: hive-1830-3.patch Carefully measure the memory usage of map side group by. Flush frequently, if the left memory is less than a threshold. mappers in group followed by joins may die OOM -- Key: HIVE-1830 URL: https://issues.apache.org/jira/browse/HIVE-1830 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Liyin Tang Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1821) describe database command
[ https://issues.apache.org/jira/browse/HIVE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang reassigned HIVE-1821: Assignee: Ning Zhang describe database command - Key: HIVE-1821 URL: https://issues.apache.org/jira/browse/HIVE-1821 Project: Hive Issue Type: New Feature Reporter: Ning Zhang Assignee: Ning Zhang a describe (extended) database command would be helpful if we introduces parameters associated with databases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1821) describe database command
[ https://issues.apache.org/jira/browse/HIVE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang updated HIVE-1821: - Issue Type: Sub-task (was: New Feature) Parent: HIVE-1820 describe database command - Key: HIVE-1821 URL: https://issues.apache.org/jira/browse/HIVE-1821 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Ning Zhang a describe (extended) database command would be helpful if we introduces parameters associated with databases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968564#action_12968564 ] Carl Steinbach commented on HIVE-1526: -- @Ashutosh: Thanks for the tip! I'll roll this into the next version of the patch (coming soon). Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.7.0 Attachments: compile.err, HIVE-1526-complete.4.patch.txt, HIVE-1526-complete.5.patch.txt, HIVE-1526-complete.6.patch.txt, HIVE-1526-no-codegen.3.patch.txt, HIVE-1526-no-codegen.4.patch.txt, HIVE-1526-no-codegen.5.patch.txt, HIVE-1526-no-codegen.6.patch.txt, HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, serde2_test.patch, svn_rm.sh, test.log, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1346) Table column name changed to _col1,_col2 ..._coln when where clause used in the select quert statement
[
https://issues.apache.org/jira/browse/HIVE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968563#action_12968563
]
Carl Steinbach commented on HIVE-1346:
--
@Vinithra: the patch needs to be rebased to trunk and submitted for review.
Another issue is that
this patch does not include any test output updates, which is almost certainly
an oversight since this
change should affect the output of EXPLAIN.
Table column name changed to _col1,_col2 ..._coln when where clause used in
the select quert statement
--
Key: HIVE-1346
URL: https://issues.apache.org/jira/browse/HIVE-1346
Project: Hive
Issue Type: Bug
Components: Clients
Affects Versions: 0.5.0, 0.6.0
Environment: ubuntu8.04, jdk-6,hive-0.5.0, hadoop-0.20.1
Reporter: Sunil Kumar
Assignee: Sunil Kumar
Priority: Minor
Attachments: HIVE-1346_patch.patch, HIVE-1346_patch.patch,
HIVE-1346_patch.patch
when where clause used in the hive query hive -ResultSetMetaData does not
give original table column name. While when where clause not used
ResultSetMetaData gives original table column names. I have used following
code:-
String tableName = user;
String sql = select * from + tableName + where
id=1;
result = stmt.executeQuery(sql);
ResultSetMetaData metaData = result.getMetaData();
int columnCount = metaData.getColumnCount();
for (int i = 1; i = columnCount; i++) {
System.out.println(Column name: +
metaData.getColumnName(i));
}
executing above code i got following result:-
Column name:_col1
Column name:_col2
while original user table columns names were (id,name).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968612#action_12968612 ] Namit Jain commented on HIVE-1763: -- +1 The approach looks fine drop table (or view) should issue warning if table doesn't exist Key: HIVE-1763 URL: https://issues.apache.org/jira/browse/HIVE-1763 Project: Hive Issue Type: Improvement Components: Metastore Reporter: dan f Assignee: Paul Butler Priority: Minor Attachments: HIVE-1763.patch drop table reports OK even if the table doesn't exist. Better to report something like mysql's Unknown table 'foo' so that, e.g., unwanted tables (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968759#action_12968759 ] Bennie Schut commented on HIVE-1675: manage to cause this with parallel=false so perhaps not such an interesting angle ;-) I've added some more logging to better understand the cause of this: 2010-12-07 15:49:44,697 INFO exec.Utilities (Utilities.java:getMapRedWork(154)) - Getting jobid:9c2eeba4-a602-4d4b-ba0b-60ce815c4ea7 from cache. 2010-12-07 15:49:44,703 INFO lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:clinit(34)) - Loaded native gpl library 2010-12-07 15:49:44,705 INFO lzo.LzoCodec (LzoCodec.java:clinit(72)) - Successfully loaded initialized native-lzo library [hadoop-lzo rev c7acdaa96a7ce04538c0716fe699ffaf11836c70] 2010-12-07 15:49:44,712 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(192)) - Total input paths to process : 1 2010-12-07 15:49:44,880 INFO exec.Utilities (Utilities.java:getMapRedWork(154)) - Getting jobid:e8b2dab2-986a-4bb1-947f-00aec5b46a06 from cache. 2010-12-07 15:49:44,882 INFO exec.ExecDriver (SessionState.java:printInfo(268)) - Job running in-process (local Hadoop) 2010-12-07 15:49:44,882 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(256)) - job_local_0001 java.lang.RuntimeException: java.io.FileNotFoundException: HIVE_PLANe8b2dab2-986a-4bb1-947f-00aec5b46a06 (No such file or directory) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:166) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:238) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:244) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:139) Caused by: java.io.FileNotFoundException: HIVE_PLANe8b2dab2-986a-4bb1-947f-00aec5b46a06 (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:106) at java.io.FileInputStream.init(FileInputStream.java:66) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:157) ... 3 more First thing I noticed which is different from a successful job would be that it's trying to get a different jobid from the cache Getting jobid:e8b2dab2-986a-4bb1-947f-00aec5b46a06 from cache I'm still confused. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1759) Many important broken links on Hive web page
[ https://issues.apache.org/jira/browse/HIVE-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo resolved HIVE-1759. --- Resolution: Fixed The issue was the content of the site MUST be all checked into svn and copied into /www/hive.apache.org, I built and committed the API docs for all older releases. Site should have no broken links 3.0 docs will propagate in the next hour or so. Many important broken links on Hive web page Key: HIVE-1759 URL: https://issues.apache.org/jira/browse/HIVE-1759 Project: Hive Issue Type: Bug Components: Documentation Reporter: Jeff Hammerbacher Assignee: Edward Capriolo The change log links are broken, perhaps because of the move to a TLP, and the Jira issue log links all point to the 0.5 issue log. Also, all of the documentation links are broken. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1823) upgrade the database thrift interface to allow parameters key-value pairs
[ https://issues.apache.org/jira/browse/HIVE-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain resolved HIVE-1823. -- Resolution: Fixed Hadoop Flags: [Reviewed] Committed. Thanks Ning upgrade the database thrift interface to allow parameters key-value pairs - Key: HIVE-1823 URL: https://issues.apache.org/jira/browse/HIVE-1823 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-1823.2.patch, HIVE-1823.patch In order to store data center specify parameters to Hive database, it is desirable to extend Hive database thrift interface with a parameters map similar to Table and Partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1727) Not able to download hive from apache site.
[ https://issues.apache.org/jira/browse/HIVE-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo resolved HIVE-1727. --- Resolution: Won't Fix That was the old location. The new location is svn co http://svn.apache.org/repos/asf/hive/trunk hive The documentation in the wki looks correct. http://wiki.apache.org/hadoop/Hive/GettingStarted#Installation_and_Configuration If you find the documentation wrong somewhere feel free to re-open.. Not able to download hive from apache site. --- Key: HIVE-1727 URL: https://issues.apache.org/jira/browse/HIVE-1727 Project: Hive Issue Type: Bug Environment: Centos 5.4 Reporter: Sangeetha Sundar Priority: Critical Original Estimate: 3h Remaining Estimate: 3h Hi , I am trying to download Hive as specified in the apache site and getting the following error. [had...@system9 ~]$ svn co http://svn.apache.org/repos/asf/hadoop/hive/trunk svn: PROPFIND request failed on '/repos/asf/hadoop/hive/trunk' svn: PROPFIND of '/repos/asf/hadoop/hive/trunk': Could not resolve hostname `svn.apache.org': Temporary failure in name resolution (http://svn.apache.org) but am able to ping that ipaddress from web browser. Please help me to resolve this issue. Or else please suggest me any other way to download hive. Thanks in advance.. -Sangita -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1728) Problem while downloading Hive from Apche site
[ https://issues.apache.org/jira/browse/HIVE-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo resolved HIVE-1728. --- Resolution: Duplicate Duplicate of HIVE-1727 Problem while downloading Hive from Apche site -- Key: HIVE-1728 URL: https://issues.apache.org/jira/browse/HIVE-1728 Project: Hive Issue Type: Bug Environment: CentOS 5.4 Reporter: Sangeetha Sundar Priority: Critical Original Estimate: 3h Remaining Estimate: 3h Hi , I am trying to download Hive as specified in the apache site and getting the following error. [had...@system9 ~]$ svn co http://svn.apache.org/repos/asf/hadoop/hive/trunk svn: PROPFIND request failed on '/repos/asf/hadoop/hive/trunk' svn: PROPFIND of '/repos/asf/hadoop/hive/trunk': Could not resolve hostname `svn.apache.org': Temporary failure in name resolution (http://svn.apache.org) but am able to ping that ipaddress from web browser. Please help me to resolve this issue. Or else please suggest me any other way to download hive. Thanks in advance.. -Sangita -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1778) simultaneously launched queries collide on hive intermediate directories
[ https://issues.apache.org/jira/browse/HIVE-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo reassigned HIVE-1778: - Assignee: Edward Capriolo simultaneously launched queries collide on hive intermediate directories Key: HIVE-1778 URL: https://issues.apache.org/jira/browse/HIVE-1778 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: Edward Capriolo we saw one instance of multiple queries for the same user launched in parallel (from a workflow engine) use the same intermediate directories. which is obviously super bad but not suprising considering how we allocate them: Random rand = new Random(); String executionId = hive_ + format.format(new Date()) + _ + Math.abs(rand.nextLong()); Java documentation says: Two Random objects created within the same millisecond will have the same sequence of random numbers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1763: - Status: Open (was: Patch Available) drop table (or view) should issue warning if table doesn't exist Key: HIVE-1763 URL: https://issues.apache.org/jira/browse/HIVE-1763 Project: Hive Issue Type: Improvement Components: Metastore Reporter: dan f Assignee: Paul Butler Priority: Minor Attachments: HIVE-1763.patch drop table reports OK even if the table doesn't exist. Better to report something like mysql's Unknown table 'foo' so that, e.g., unwanted tables (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968835#action_12968835 ] Namit Jain commented on HIVE-1763: -- However, it will need a lot of test result files to be updated. Most of the tests will break drop table (or view) should issue warning if table doesn't exist Key: HIVE-1763 URL: https://issues.apache.org/jira/browse/HIVE-1763 Project: Hive Issue Type: Improvement Components: Metastore Reporter: dan f Assignee: Paul Butler Priority: Minor Attachments: HIVE-1763.patch drop table reports OK even if the table doesn't exist. Better to report something like mysql's Unknown table 'foo' so that, e.g., unwanted tables (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1303) Adding/selecting many external partitions tables in one session eventually fails
[
https://issues.apache.org/jira/browse/HIVE-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward Capriolo resolved HIVE-1303.
---
Resolution: Won't Fix
This was solved by doing pooling at the JPOX/Data Nucleus level.
Adding/selecting many external partitions tables in one session eventually
fails
Key: HIVE-1303
URL: https://issues.apache.org/jira/browse/HIVE-1303
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Edward Capriolo
Priority: Critical
echo create external table if not exists edtest ( dat string ) partitioned
by (dummy string) location '/tmp/a'; test.q
for i in {1..3000} ; do echo alter table ed_test add partition
(dummy='${i}') location '/tmp/duh'; ; done test.q
hive -f test.q
Also, there are problems working with this type of table as well. :(
$ hive -e explain select * from X_action
Hive history file=/tmp/XX/hive_job_log_media6_201004121029_170696698.txt
FAILED: Error in semantic analysis: javax.jdo.JDODataStoreException: Access
denied for user 'hivadm'@'XX' (using password: YES)
NestedThrowables:
java.sql.SQLException: Access denied for user 'hivadm'@'XX' (using
password: YES)
Interestingly enough if we specify some partitions we can dodge this error. I
get the fealing that the select * is trying to select too many partitions and
causing this error.
2010-04-12 10:33:02,789 ERROR metadata.Hive (Hive.java:getPartition(629)) -
javax.jdo.JDODataStoreException: Access denied for user 'hivadm'@'rs01
.sd.pl.pvt' (using password: YES)
at
org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)
at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:274)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:551)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:716)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:704)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:593)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:418)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:620)
at
org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:215)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:4883)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5224)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:44)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
NestedThrowablesStackTrace:
java.sql.SQLException: Access denied for user
'hivadm'@'X.domain.whatetever' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3436)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1247)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2775)
at com.mysql.jdbc.Connection.init(Connection.java:1555)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
at
org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:142)
at
org.datanucleus.store.rdbms.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:118)
at
org.datanucleus.store.rdbms.ConnectionProviderPriorityList.getConnection(ConnectionProviderPriorityList.java:59)
at
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968841#action_12968841 ] Namit Jain commented on HIVE-1648: -- @Yongqiang, you have missed the test changes in the patch - can you add them also ? Automatically gathering stats when reading a table/partition Key: HIVE-1648 URL: https://issues.apache.org/jira/browse/HIVE-1648 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Paul Butler Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, HIVE-1648.patch, hive-1648.svn.patch HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to gathering stats. This requires additional scan of the data. Stats gathering can be piggy-backed on TableScanOperator whenever a table/partition is scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1508) Add cleanup method to HiveHistory class
[ https://issues.apache.org/jira/browse/HIVE-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968845#action_12968845 ] Namit Jain commented on HIVE-1508: -- +1 Add cleanup method to HiveHistory class --- Key: HIVE-1508 URL: https://issues.apache.org/jira/browse/HIVE-1508 Project: Hive Issue Type: Bug Components: Metastore Reporter: Anurag Phadke Assignee: Edward Capriolo Priority: Blocker Fix For: 0.7.0 Attachments: hive-1508-1-patch.txt Running hive server for long time 90 minutes results in too many open file-handles, eventually causing the server to crash as the server runs out of file handle. Actual bug as described by Carl Steinbach: the hive_job_log_* files are created by the HiveHistory class. This class creates a PrintWriter for writing to the file, but never closes the writer. It looks like we need to add a cleanup method to HiveHistory that closes the PrintWriter and does any other necessary cleanup. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968855#action_12968855 ] John Sichi commented on HIVE-1763: -- See HIVE-1542 for my suggested approach. drop table (or view) should issue warning if table doesn't exist Key: HIVE-1763 URL: https://issues.apache.org/jira/browse/HIVE-1763 Project: Hive Issue Type: Improvement Components: Metastore Reporter: dan f Assignee: Paul Butler Priority: Minor Attachments: HIVE-1763.patch drop table reports OK even if the table doesn't exist. Better to report something like mysql's Unknown table 'foo' so that, e.g., unwanted tables (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1778) simultaneously launched queries collide on hive intermediate directories
[ https://issues.apache.org/jira/browse/HIVE-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968859#action_12968859 ] Joydeep Sen Sarma commented on HIVE-1778: - whatever works - we could pass in hash the query string and time (perhaps a nanosecond timer) to come up with a better seed for the random generator for example. simultaneously launched queries collide on hive intermediate directories Key: HIVE-1778 URL: https://issues.apache.org/jira/browse/HIVE-1778 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: Edward Capriolo we saw one instance of multiple queries for the same user launched in parallel (from a workflow engine) use the same intermediate directories. which is obviously super bad but not suprising considering how we allocate them: Random rand = new Random(); String executionId = hive_ + format.format(new Date()) + _ + Math.abs(rand.nextLong()); Java documentation says: Two Random objects created within the same millisecond will have the same sequence of random numbers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1834) more debugging for locking
[ https://issues.apache.org/jira/browse/HIVE-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1834: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed! Thanks Namit! more debugging for locking -- Key: HIVE-1834 URL: https://issues.apache.org/jira/browse/HIVE-1834 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1834.1.patch Along with the time and the queryid, it might be a good idea to log if the lock was acquired explicitly (by a lock command) or implicitly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-97) tab completion for hive cli
[ https://issues.apache.org/jira/browse/HIVE-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo resolved HIVE-97. - Resolution: Duplicate This is solved and is being enhanced by. HIVE-1835. tab completion for hive cli --- Key: HIVE-97 URL: https://issues.apache.org/jira/browse/HIVE-97 Project: Hive Issue Type: Improvement Components: Clients, Documentation Reporter: Pete Wyckoff jline provides a framework for implementing tab completion. if one can somehow enumerate the grammar in a way that jline understands, this would improve usability a lot. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1838) Add quickLZ compression codec for Hive.
Add quickLZ compression codec for Hive. --- Key: HIVE-1838 URL: https://issues.apache.org/jira/browse/HIVE-1838 Project: Hive Issue Type: New Feature Reporter: He Yongqiang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1835) Better auto-complete for Hive
[ https://issues.apache.org/jira/browse/HIVE-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Butler updated HIVE-1835: -- Attachment: HIVE-1835.2.patch Fixed missing file Better auto-complete for Hive - Key: HIVE-1835 URL: https://issues.apache.org/jira/browse/HIVE-1835 Project: Hive Issue Type: New Feature Components: CLI Reporter: Paul Butler Assignee: Paul Butler Priority: Minor Attachments: HIVE-1835.2.patch, HIVE-1835.patch - Add functions and keywords to auto-complete list - Make Hive auto-complete aware of Hive delimiters (eg. whitespace, parentheses) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1837) optional timeout for hive clients
[ https://issues.apache.org/jira/browse/HIVE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1837: --- Attachment: hive-1837.1.patch an initial patch. will do more tests in our env. optional timeout for hive clients - Key: HIVE-1837 URL: https://issues.apache.org/jira/browse/HIVE-1837 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: He Yongqiang Attachments: hive-1837.1.patch It would be a good idea to have a optional timeout for hive clients. We encountered a query today, which seemed to have run by mistake, and it was running for about a month. This was holding zookeeper locks, and making the whole debugging more complex than it should be. It would be a good idea to have a timeout for a hive client. @Ning, I remember there was some issue with the Hive client having a timeout of 1 day with HiPal. Do you remember the details ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1839) Error message for Both Left and Right Aliases Encountered in Join time cites wrong row/col
Error message for Both Left and Right Aliases Encountered in Join time cites wrong row/col Key: HIVE-1839 URL: https://issues.apache.org/jira/browse/HIVE-1839 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Adam Kramer In all cases of the above error, the error message looks like this: FAILED: Error in semantic analysis: line 0:-1 Both Left and Right Aliases Encountered in Join time ...the 0:-1 is incorrect. This should provide the row and the column number. Ideally, it would also provide the textual left and right aliases so that the user could identify which aliases are encountered where since this is rarely obvious. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968907#action_12968907 ] Ning Zhang commented on HIVE-1526: -- Thanks Ashutosh and Carl! The changes look good and all unit tests have passed. However, there are conflicts after another JIRA is committed. Carl, can you pelase regenerate the patch yet another time? I'll try my best to test and commit ASAP to avoid conflicts again. Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.7.0 Attachments: compile.err, HIVE-1526-complete.4.patch.txt, HIVE-1526-complete.5.patch.txt, HIVE-1526-complete.6.patch.txt, HIVE-1526-complete.7.patch.txt, HIVE-1526-no-codegen.3.patch.txt, HIVE-1526-no-codegen.4.patch.txt, HIVE-1526-no-codegen.5.patch.txt, HIVE-1526-no-codegen.6.patch.txt, HIVE-1526-no-codegen.7.patch.txt, HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, serde2_test.patch, svn_rm.sh, test.log, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1836) Extend the CREATE DATABASE command with DBPROPERTIES
[
https://issues.apache.org/jira/browse/HIVE-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ning Zhang updated HIVE-1836:
-
Attachment: HIVE-1836.patch
Attaching HIVE-1836.patch that addresses both HIVE-1821 (DESC DATABASE) and
1836 (CREATE DATABASE WITH DBPROPERTIES).
Extend the CREATE DATABASE command with DBPROPERTIES
Key: HIVE-1836
URL: https://issues.apache.org/jira/browse/HIVE-1836
Project: Hive
Issue Type: Sub-task
Reporter: Ning Zhang
Assignee: Ning Zhang
Attachments: HIVE-1836.patch
We should be able to assign key-value pairs of properties to Hive databases.
The proposed syntax is similar to the CREATE TABLE and CREATE INDEX commands:
{code}
CREATE DATABASE DB_NAME WITH DBPROPERTIES ('key1' = 'value1', 'key2' =
'value2');
{code}
The
{code}
DESC DATABASE EXTENDED DB_NAME;
{code}
should be able to display the properties. (requires HIVE-1821)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1836) Extend the CREATE DATABASE command with DBPROPERTIES
[
https://issues.apache.org/jira/browse/HIVE-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ning Zhang updated HIVE-1836:
-
Status: Patch Available (was: Open)
Extend the CREATE DATABASE command with DBPROPERTIES
Key: HIVE-1836
URL: https://issues.apache.org/jira/browse/HIVE-1836
Project: Hive
Issue Type: Sub-task
Reporter: Ning Zhang
Assignee: Ning Zhang
Attachments: HIVE-1836.patch
We should be able to assign key-value pairs of properties to Hive databases.
The proposed syntax is similar to the CREATE TABLE and CREATE INDEX commands:
{code}
CREATE DATABASE DB_NAME WITH DBPROPERTIES ('key1' = 'value1', 'key2' =
'value2');
{code}
The
{code}
DESC DATABASE EXTENDED DB_NAME;
{code}
should be able to display the properties. (requires HIVE-1821)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1821) describe database command
[ https://issues.apache.org/jira/browse/HIVE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968915#action_12968915 ] Ning Zhang commented on HIVE-1821: -- a patch is uploaded to HIVE-1836 that address this patch. describe database command - Key: HIVE-1821 URL: https://issues.apache.org/jira/browse/HIVE-1821 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Ning Zhang a describe (extended) database command would be helpful if we introduces parameters associated with databases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1837) optional timeout for hive clients
[ https://issues.apache.org/jira/browse/HIVE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1837: --- Attachment: hive-1837.2.patch a new patch after some tests in the cluster optional timeout for hive clients - Key: HIVE-1837 URL: https://issues.apache.org/jira/browse/HIVE-1837 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: He Yongqiang Attachments: hive-1837.1.patch, hive-1837.2.patch It would be a good idea to have a optional timeout for hive clients. We encountered a query today, which seemed to have run by mistake, and it was running for about a month. This was holding zookeeper locks, and making the whole debugging more complex than it should be. It would be a good idea to have a timeout for a hive client. @Ning, I remember there was some issue with the Hive client having a timeout of 1 day with HiPal. Do you remember the details ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1821) describe database command
[ https://issues.apache.org/jira/browse/HIVE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain resolved HIVE-1821. -- Resolution: Duplicate Duplicate of HIVE-1836 describe database command - Key: HIVE-1821 URL: https://issues.apache.org/jira/browse/HIVE-1821 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Ning Zhang a describe (extended) database command would be helpful if we introduces parameters associated with databases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1821) describe database command
[ https://issues.apache.org/jira/browse/HIVE-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968939#action_12968939 ] Namit Jain commented on HIVE-1821: -- If you are doing this, do you want to add a 'alter database' also ? describe database command - Key: HIVE-1821 URL: https://issues.apache.org/jira/browse/HIVE-1821 Project: Hive Issue Type: Sub-task Reporter: Ning Zhang Assignee: Ning Zhang a describe (extended) database command would be helpful if we introduces parameters associated with databases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1836) Extend the CREATE DATABASE command with DBPROPERTIES
[
https://issues.apache.org/jira/browse/HIVE-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968961#action_12968961
]
Ning Zhang commented on HIVE-1836:
--
Yes, will add 'alter database' also in a follow-up JIRA. A question is that if
alter the current database, wether to change the HiveConf parameters
accordingly. Since 'alter database' is not a blocking issue yet, I'm working on
HIVE-1820 first and then come back to that.
Extend the CREATE DATABASE command with DBPROPERTIES
Key: HIVE-1836
URL: https://issues.apache.org/jira/browse/HIVE-1836
Project: Hive
Issue Type: Sub-task
Reporter: Ning Zhang
Assignee: Ning Zhang
Attachments: HIVE-1836.patch
We should be able to assign key-value pairs of properties to Hive databases.
The proposed syntax is similar to the CREATE TABLE and CREATE INDEX commands:
{code}
CREATE DATABASE DB_NAME WITH DBPROPERTIES ('key1' = 'value1', 'key2' =
'value2');
{code}
The
{code}
DESC DATABASE EXTENDED DB_NAME;
{code}
should be able to display the properties. (requires HIVE-1821)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1837) optional timeout for hive clients
[ https://issues.apache.org/jira/browse/HIVE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969000#action_12969000 ] Ashutosh Chauhan commented on HIVE-1837: You get this feature for free when you move to secure Hadoop. A mapreduce job by default gets a token which expires in 24 hrs. So, usually MR job spawned by Hive query will fail after that time. Job may request renewal upto 7 days. Beyond that, special provisions are required. So, timeout is inherently built into secure hadoop. optional timeout for hive clients - Key: HIVE-1837 URL: https://issues.apache.org/jira/browse/HIVE-1837 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: He Yongqiang Attachments: hive-1837.1.patch, hive-1837.2.patch It would be a good idea to have a optional timeout for hive clients. We encountered a query today, which seemed to have run by mistake, and it was running for about a month. This was holding zookeeper locks, and making the whole debugging more complex than it should be. It would be a good idea to have a timeout for a hive client. @Ning, I remember there was some issue with the Hive client having a timeout of 1 day with HiPal. Do you remember the details ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1836) Extend the CREATE DATABASE command with DBPROPERTIES
[
https://issues.apache.org/jira/browse/HIVE-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969037#action_12969037
]
Namit Jain commented on HIVE-1836:
--
+1
Extend the CREATE DATABASE command with DBPROPERTIES
Key: HIVE-1836
URL: https://issues.apache.org/jira/browse/HIVE-1836
Project: Hive
Issue Type: Sub-task
Reporter: Ning Zhang
Assignee: Ning Zhang
Attachments: HIVE-1836.patch
We should be able to assign key-value pairs of properties to Hive databases.
The proposed syntax is similar to the CREATE TABLE and CREATE INDEX commands:
{code}
CREATE DATABASE DB_NAME WITH DBPROPERTIES ('key1' = 'value1', 'key2' =
'value2');
{code}
The
{code}
DESC DATABASE EXTENDED DB_NAME;
{code}
should be able to display the properties. (requires HIVE-1821)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1096) Hive Variables
[
https://issues.apache.org/jira/browse/HIVE-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969089#action_12969089
]
Namit Jain commented on HIVE-1096:
--
sure, that would be very useful
Let me know if you run into any issues
Hive Variables
--
Key: HIVE-1096
URL: https://issues.apache.org/jira/browse/HIVE-1096
Project: Hive
Issue Type: New Feature
Components: Query Processor
Reporter: Edward Capriolo
Assignee: Edward Capriolo
Fix For: 0.7.0
Attachments: 1096-9.diff, hive-1096-10-patch.txt,
hive-1096-11-patch.txt, hive-1096-12.patch.txt, hive-1096-15.patch.txt,
hive-1096-15.patch.txt, hive-1096-2.diff, hive-1096-20.patch.txt,
hive-1096-7.diff, hive-1096-8.diff, hive-1096.diff
From mailing list:
--Amazon Elastic MapReduce version of Hive seems to have a nice feature
called Variables. Basically you can define a variable via command-line
while invoking hive with -d DT=2009-12-09 and then refer to the variable via
${DT} within the hive queries. This could be extremely useful. I can't seem
to find this feature even on trunk. Is this feature currently anywhere in the
roadmap?--
This could be implemented in many places.
A simple place to put this is
in Driver.compile or Driver.run we can do string substitutions at that level,
and further downstream need not be effected.
There could be some benefits to doing this further downstream, parser,plan.
but based on the simple needs we may not need to overthink this.
I will get started on implementing in compile unless someone wants to discuss
this more.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1837) optional timeout for hive clients
[ https://issues.apache.org/jira/browse/HIVE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969093#action_12969093 ] Namit Jain commented on HIVE-1837: -- @Ashutosh, we cant wait for this feature till secure hadoop is available. Once Hive is migrated to that, we can change the implementation of this feature. @Yongqiang, can you add the new parameter definition in hive-default.xml ? Also, can you make the thread sleep time (10 min.) configurable ? Can you add a new test for the same - I mean, have a very small timeout and thread sleep time, and a custom script which is sleeping indefinitely ? optional timeout for hive clients - Key: HIVE-1837 URL: https://issues.apache.org/jira/browse/HIVE-1837 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: He Yongqiang Attachments: hive-1837.1.patch, hive-1837.2.patch It would be a good idea to have a optional timeout for hive clients. We encountered a query today, which seemed to have run by mistake, and it was running for about a month. This was holding zookeeper locks, and making the whole debugging more complex than it should be. It would be a good idea to have a timeout for a hive client. @Ning, I remember there was some issue with the Hive client having a timeout of 1 day with HiPal. Do you remember the details ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1415) add CLI command for executing a SQL script
[ https://issues.apache.org/jira/browse/HIVE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo reassigned HIVE-1415: - Assignee: Edward Capriolo add CLI command for executing a SQL script -- Key: HIVE-1415 URL: https://issues.apache.org/jira/browse/HIVE-1415 Project: Hive Issue Type: Improvement Components: Clients Affects Versions: 0.5.0 Reporter: John Sichi Assignee: Edward Capriolo Fix For: 0.7.0 Attachments: hive-1415-1-patch.txt Suggestion in HIVE-1405 was source, e.g. source somescript.sql; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1837) optional timeout for hive clients
[ https://issues.apache.org/jira/browse/HIVE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969141#action_12969141 ] He Yongqiang commented on HIVE-1837: it is still very difficult to add a testcase. It's because there is a System.exit(-1) in the monitor thread. The test process will exit. optional timeout for hive clients - Key: HIVE-1837 URL: https://issues.apache.org/jira/browse/HIVE-1837 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: He Yongqiang Attachments: hive-1837.1.patch, hive-1837.2.patch It would be a good idea to have a optional timeout for hive clients. We encountered a query today, which seemed to have run by mistake, and it was running for about a month. This was holding zookeeper locks, and making the whole debugging more complex than it should be. It would be a good idea to have a timeout for a hive client. @Ning, I remember there was some issue with the Hive client having a timeout of 1 day with HiPal. Do you remember the details ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1838) Add quickLZ compression codec for Hive.
[ https://issues.apache.org/jira/browse/HIVE-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969144#action_12969144 ] He Yongqiang commented on HIVE-1838: No. I mean compression codec for Hive. It could be used to compress intermediate data. Here are some results: 5. Hadoop compression with native library (COMPRESSLEVEL=BEST_SPEED) time java -Djava.library.path=/data/users/heyongqiang/hadoop-0.20/build/native/Linux-amd64-64/lib/ CompressFile real0m34.179s user0m29.031s sys 0m1.607s compressed size: 275M 6. LZF [heyongqi...@dev782 compress_test]$ time lzf -c 00_0 real0m39.031s user0m8.727s sys 0m2.231s compressed size: 393M 7. FastLZ time fastlz/6pack -1 00_0 00_0.fastlz real0m19.020s user0m18.083s sys 0m0.935s compressed size: 391M 8.QuickLZ time ./compress_file ../00_0 ../00_0.quicklz real0m15.652s user0m14.047s sys 0m1.603s compressed size: 334M I modified QuickLZ's compress_file code to use a buffer for fairness. It turns out the result is very close to FastLZ. The modified version of QuickLZ is just one second better. Add quickLZ compression codec for Hive. --- Key: HIVE-1838 URL: https://issues.apache.org/jira/browse/HIVE-1838 Project: Hive Issue Type: New Feature Reporter: He Yongqiang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1837) optional timeout for hive clients
[ https://issues.apache.org/jira/browse/HIVE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969146#action_12969146 ] Ashutosh Chauhan commented on HIVE-1837: One way to get around System.exit() problem in testcase is to create your own SecurityManager and use that. In your SecurityManager override checkExit() and throw an exception. This way whenever System.exit() is encountered, an exception will be thrown. In your testcase you can catch the exception and then do the asserts that you want. I did very similar things while writing junit tests for Howl. optional timeout for hive clients - Key: HIVE-1837 URL: https://issues.apache.org/jira/browse/HIVE-1837 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: He Yongqiang Attachments: hive-1837.1.patch, hive-1837.2.patch It would be a good idea to have a optional timeout for hive clients. We encountered a query today, which seemed to have run by mistake, and it was running for about a month. This was holding zookeeper locks, and making the whole debugging more complex than it should be. It would be a good idea to have a timeout for a hive client. @Ning, I remember there was some issue with the Hive client having a timeout of 1 day with HiPal. Do you remember the details ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1837) optional timeout for hive clients
[ https://issues.apache.org/jira/browse/HIVE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12969151#action_12969151 ] He Yongqiang commented on HIVE-1837: Cool, Thanks Ashutosh! I will try that. optional timeout for hive clients - Key: HIVE-1837 URL: https://issues.apache.org/jira/browse/HIVE-1837 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: He Yongqiang Attachments: hive-1837.1.patch, hive-1837.2.patch It would be a good idea to have a optional timeout for hive clients. We encountered a query today, which seemed to have run by mistake, and it was running for about a month. This was holding zookeeper locks, and making the whole debugging more complex than it should be. It would be a good idea to have a timeout for a hive client. @Ning, I remember there was some issue with the Hive client having a timeout of 1 day with HiPal. Do you remember the details ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
