[jira] [Commented] (HIVE-2945) Hive Client is hanging since MRApp Master doesnt exist after the task is finished
[ https://issues.apache.org/jira/browse/HIVE-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251616#comment-13251616 ] Edward Capriolo commented on HIVE-2945: --- I am not sure anyone has tried hive in the Mapred.NEXT setup yet. I do not know enough about the infrastructure to comment definitively , but it sounds like it would be nice if the processes hung around for a few moments after completion so Hive client could check in to them. Strange though because you would not expect a hive job to hang around long then a typical map/reduce job. It would be great if you can give a stack trace and find what point in the hive code is hanging. Hive Client is hanging since MRApp Master doesnt exist after the task is finished -- Key: HIVE-2945 URL: https://issues.apache.org/jira/browse/HIVE-2945 Project: Hive Issue Type: Bug Components: Clients Reporter: Kristam Subba Swathi Submit a hive query which uses mapred to give the result(Say join query) If we submit a job to resourceManager process. it will create a MRAppMaster process to excute this task . MRAppMaster will create YarnChild process to excute Map task or reduce task. after task is finished,resourceManager will kill MRAppMaster and YarnChild process. When the hive client is about to get the progress from MRAppMaster ,the task is already finished ,so MRAppMaster process don't exist. so the client will try to connect it for ever. that's why hive client hangs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Looking at the columns table
Hey all. Our metastore in mysql is fairly large over 12GB. All the storage here is the columns table. It seems that each column is stored for each partition/storage descriptor as a one-many relationship. In our case all the partitions have the same column definition. My thinking. Should the relationship from columns-partition/storage descriptor be a many-many? In this way we only store the column once and the current column table can reference the primary key of this column. This should bring the size of this table down really drastically. Since every other table in the metastore is so small this huge columns table looks like the only scalability choke point we have. Edward
Re: Looking at the columns table
Hey Ed, Your thinking is correct and has been implemented in https://issues.apache.org/jira/browse/HIVE-2246 Time to upgrade to 0.8 :) Thanks, Ashutosh On Wed, Apr 11, 2012 at 07:53, Edward Capriolo edlinuxg...@gmail.comwrote: Hey all. Our metastore in mysql is fairly large over 12GB. All the storage here is the columns table. It seems that each column is stored for each partition/storage descriptor as a one-many relationship. In our case all the partitions have the same column definition. My thinking. Should the relationship from columns-partition/storage descriptor be a many-many? In this way we only store the column once and the current column table can reference the primary key of this column. This should bring the size of this table down really drastically. Since every other table in the metastore is so small this huge columns table looks like the only scalability choke point we have. Edward
[jira] [Updated] (HIVE-2942) substr on string containing UTF-8 characters produces StringIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HIVE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2942: Status: Patch Available (was: Open) substr on string containing UTF-8 characters produces StringIndexOutOfBoundsException - Key: HIVE-2942 URL: https://issues.apache.org/jira/browse/HIVE-2942 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2942.D2727.1.patch After HIVE-2792, the substr function produces a StringIndexOutOfBoundsException when called on a string containing UTF-8 characters without the length argument being present. E.g. select substr(str, 1) from table1; now fails with that exception if str contains a UTF-8 character for any row in the table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2942) substr on string containing UTF-8 characters produces StringIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HIVE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251744#comment-13251744 ] Phabricator commented on HIVE-2942: --- kevinwilfong has commented on the revision HIVE-2942 [jira] substr on string containing UTF-8 characters produces StringIndexOutOfBoundsException. Verified all the tests still pass. REVISION DETAIL https://reviews.facebook.net/D2727 substr on string containing UTF-8 characters produces StringIndexOutOfBoundsException - Key: HIVE-2942 URL: https://issues.apache.org/jira/browse/HIVE-2942 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2942.D2727.1.patch After HIVE-2792, the substr function produces a StringIndexOutOfBoundsException when called on a string containing UTF-8 characters without the length argument being present. E.g. select substr(str, 1) from table1; now fails with that exception if str contains a UTF-8 character for any row in the table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2937) TestHiveServerSessions hangs when executed directly
[
https://issues.apache.org/jira/browse/HIVE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251808#comment-13251808
]
Kevin Wilfong commented on HIVE-2937:
-
I ran into this issue too. It seems to be caused by creating the HiveClient's
too quickly. The initialization of the HiveClient initializes the
HiveServerHandler which initializes the HMSHandler. The initializations of the
HMSHandler's happen in such quick succession that when the second call to
create the default db occurs, the first call hasn't finished creating the db
yet so it attempts to create the same db and never gets out of this method
(possibly a derby issue).
It could also be fixed by adding a Thread.sleep between creating HiveClient's,
but Navis's solution seems much more appropriate.
@Ashutosh I don't think it is related to either of those patches as previous
builds appear to fail for the same reason
https://builds.apache.org/job/Hive-trunk-h0.21/1355/ also both of those patches
should have only affected map reduce jobs, not the metastore.
TestHiveServerSessions hangs when executed directly
---
Key: HIVE-2937
URL: https://issues.apache.org/jira/browse/HIVE-2937
Project: Hive
Issue Type: Test
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-2937.D2697.1.patch
{code}
ant test -Doffline=true -Dtestcase=TestHiveServerSessions
{code}
Hangs infinitely.
I couldn't imagine exact cause of the problem, but found that by adding 'new
HiveServer.HiveServerHandler();' in setup(), test resulted to success.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2937) TestHiveServerSessions hangs when executed directly
[
https://issues.apache.org/jira/browse/HIVE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251812#comment-13251812
]
Phabricator commented on HIVE-2937:
---
kevinwilfong has accepted the revision HIVE-2937 [jira] TestHiveServerSessions
hangs when executed directly.
+1 running tests
REVISION DETAIL
https://reviews.facebook.net/D2697
BRANCH
DPAL-1079
TestHiveServerSessions hangs when executed directly
---
Key: HIVE-2937
URL: https://issues.apache.org/jira/browse/HIVE-2937
Project: Hive
Issue Type: Test
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-2937.D2697.1.patch
{code}
ant test -Doffline=true -Dtestcase=TestHiveServerSessions
{code}
Hangs infinitely.
I couldn't imagine exact cause of the problem, but found that by adding 'new
HiveServer.HiveServerHandler();' in setup(), test resulted to success.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Problems with Arc/Phabricator
If we are going to switch from fabricator we just might as well go
back to not using anything. Review board was really clunky and
confusing.
Edward
On Tue, Apr 10, 2012 at 9:27 PM, Kevin Wilfong kevinwilf...@fb.com wrote:
+John Sichi as he did a lot of the work on getting Hive to use Phabricator.
Is there anything preventing people from using Review Board for patches now,
except maybe convention? As for applying patches, in either case, you can
download the patch from the JIRA site.
To be clear, I am not trying to aggressively force Phabricator on Hive, these
are just suggestions.
From: Carl Steinbach c...@cloudera.commailto:c...@cloudera.com
Date: Tue, 10 Apr 2012 18:10:44 -0700
To: Microsoft Office User kevinwilf...@fb.commailto:kevinwilf...@fb.com
Cc: dev@hive.apache.orgmailto:dev@hive.apache.org
dev@hive.apache.orgmailto:dev@hive.apache.org
Subject: Re: Problems with Arc/Phabricator
Ok, I'll submit an issue. But then what? Who is going to fix it? Perhaps in
the meantime we should just switch back to using reviewboard while we wait
for Arc and Phabricator to mature as open source projects.
On Tue, Apr 10, 2012 at 5:58 PM, Kevin Wilfong
kevinwilf...@fb.commailto:kevinwilf...@fb.com wrote:
There's a list of ways to report issues/get help on the side bar of this
site http://phabricator.org/
It suggests #phabricator
irc://chat.freenode.net/phabricatorhttp://chat.freenode.net/phabricator
as the best
place to get help with issues.
On 4/10/12 5:42 PM, Carl Steinbach
c...@cloudera.commailto:c...@cloudera.com wrote:
Arc won't let me apply the patch for HIVE-2646. I've also noticed some
other problems (e.g. missing TEST PLAN in checkin comments), and was
wondering how we can get these problems resolved? Since most of this work
was done at Facebook, who is the designated point of contact for
Arc/Phabricator issues going forward?
Here's the error I get when applying HIVE-2646:
% arc patch D2133 --trace
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive-test/.arc_jira_lib'...
[0] exec $ (cd '/Users/carl/Work/repos/hive-test'; git rev-parse
--show-cdup)
[0] exec 13,599 us
[1] conduit differential.getdiff()
[1] conduit 608,237 us
[2] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git apply
--index
--reject)
[2] exec 13,178 us
Fatal error: Uncaught exception 'CommandException' with message 'Command
'(cd '/Users/carl/Work/repos/hive-test/'; git apply --index --reject)'
failed with error #128:
stdout:
stderr:
stdin:10: trailing whitespace.
stdin:111: trailing whitespace.
fileset dir=${build.ivy.lib.dir}/default includes=*.jar
fatal: corrupt patch at line 609
' in /Users/carl/.local/pkg/libphutil/src/future/exec/ExecFuture.php:346
Stack trace:
#0
/Users/carl/.local/pkg/arcanist/src/workflow/patch/ArcanistPatchWorkflow.p
hp(412):
ExecFuture-resolvex()
#1 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(266):
ArcanistPatchWorkflow-run()
#2 {main}
thrown in
/Users/carl/.local/pkg/libphutil/src/future/exec/ExecFuture.php
on line 346
Thanks.
Carl
Re: Problems with Arc/Phabricator
On Wed, Apr 11, 2012 at 11:48 AM, Edward Capriolo edlinuxg...@gmail.com wrote: If we are going to switch from fabricator we just might as well go back to not using anything. Review board was really clunky and confusing. I'm mostly +1 to that. If no one is supporting phabricator, then it won't work for long. Personally, I'd love it if we could move Hive to git completely. Has anyone used gerrit? The videos of it make it look better than sliced bread. -- Owen
Re: Problems with Arc/Phabricator
+1 on moving away from arc/phabricator. It works great when it works, but most of the time it doesnt work. Ashutosh On Wed, Apr 11, 2012 at 11:57, Owen O'Malley omal...@apache.org wrote: On Wed, Apr 11, 2012 at 11:48 AM, Edward Capriolo edlinuxg...@gmail.com wrote: If we are going to switch from fabricator we just might as well go back to not using anything. Review board was really clunky and confusing. I'm mostly +1 to that. If no one is supporting phabricator, then it won't work for long. Personally, I'd love it if we could move Hive to git completely. Has anyone used gerrit? The videos of it make it look better than sliced bread. -- Owen
[jira] [Work started] (HIVE-895) Add SerDe for Avro serialized data
[ https://issues.apache.org/jira/browse/HIVE-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-895 started by Jakob Homan. Add SerDe for Avro serialized data -- Key: HIVE-895 URL: https://issues.apache.org/jira/browse/HIVE-895 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Reporter: Jeff Hammerbacher Assignee: Jakob Homan As Avro continues to mature, having a SerDe to allow HiveQL queries over Avro data seems like a solid win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-0.8.1-SNAPSHOT-h0.21 - Build # 250 - Fixed
Changes for Build #247 Changes for Build #248 Changes for Build #249 Changes for Build #250 All tests passed The Apache Jenkins build system has built Hive-0.8.1-SNAPSHOT-h0.21 (build #250) Status: Fixed Check console output at https://builds.apache.org/job/Hive-0.8.1-SNAPSHOT-h0.21/250/ to view the results.
Hive-trunk-h0.21 - Build # 1368 - Fixed
Changes for Build #1365 [hashutosh] Upping the version number from 0.9 to 0.10 on trunk. Changes for Build #1366 Changes for Build #1367 [kevinwilfong] HIVE-2907. Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) Changes for Build #1368 All tests passed The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1368) Status: Fixed Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1368/ to view the results.
[jira] [Updated] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-2928: --- Attachment: (was: HIVE-2928.patch) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2946) Hive metastore does not have any log messages while shutting itself down.
Hive metastore does not have any log messages while shutting itself down. -- Key: HIVE-2946 URL: https://issues.apache.org/jira/browse/HIVE-2946 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor When hive metastore is started, the event is logged with good amount of information. But when it is stopped ( using a kill) command, no information is being written to the logs. It will be good if we can add a shutdown event to the HiveMetastore class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-2928: --- Attachment: HIVE-2928.patch Updated to remove package.jdo change. Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2405) get_privilege does not get user level privilege
[
https://issues.apache.org/jira/browse/HIVE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-2405:
-
Component/s: Security
Authorization
get_privilege does not get user level privilege
---
Key: HIVE-2405
URL: https://issues.apache.org/jira/browse/HIVE-2405
Project: Hive
Issue Type: Bug
Components: Authorization, Security
Reporter: He Yongqiang
Assignee: He Yongqiang
Fix For: 0.8.0
Attachments: HIVE-2405.1.patch, HIVE-2405.2.patch
hive set hive.security.authorization.enabled=true;
hive grant all to user heyongqiang;
hive show grant user heyongqiang;
principalName heyongqiang
principalType USER
privilege All
grantTime Wed Aug 24 11:51:54 PDT 2011
grantor heyongqiang
Time taken: 0.032 seconds
hive CREATE TABLE src (foo INT, bar STRING);
Authorization failed:No privilege 'Create' found for outputs {
database:default}. Use show grant to get more details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251965#comment-13251965 ] Mithun Radhakrishnan commented on HIVE-2928: @Carl: Thanks. I'd seen that thread before, but I hadn't actually tested that out myself. I just did, and here's what I found out: 1. If the complete schema is created apriori (using the hive-schema-0.9.0.oracle.sql in the attached patch), then hive works with Oracle. (This is with datanucleus.validateColumns = false.) 2. What's neat is that datanucleus.autoCreateSchema = true doesn't mess this up, because the schema is completely constructed. The exception in the Original Description was a result of there being schema differences that JDO attempted to resolve. (For the record, Datanucleus does recommend turning both flags off, for the sake of performance. They're meant to be used for the first start-up.) The offshoot of this would be that any changes in the schema would have to be resolved using a migration-script for Oracle, and won't be done automatically by the JDO-lib. I've modified the attached patch to remove the proposed package.jdo change, and keep just the Oracle schema-sql script. Thanks a bunch. Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2946) Hive metastore does not have any log messages while shutting itself down.
[ https://issues.apache.org/jira/browse/HIVE-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2946: -- Attachment: HIVE-2946.D2745.1.patch avandana requested code review of HIVE-2946 [jira] Hive metastore does not have any log messages while shutting itself down.. Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2946 Added shutdown hook to log message while shutting down metastore. When hive metastore is started, the event is logged with good amount of information. But when it is stopped ( using a kill) command, no information is being written to the logs. It will be good if we can add a shutdown event to the HiveMetastore class. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2745 AFFECTED FILES metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6273/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Hive metastore does not have any log messages while shutting itself down. -- Key: HIVE-2946 URL: https://issues.apache.org/jira/browse/HIVE-2946 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Labels: metastore Attachments: HIVE-2946.D2745.1.patch When hive metastore is started, the event is logged with good amount of information. But when it is stopped ( using a kill) command, no information is being written to the logs. It will be good if we can add a shutdown event to the HiveMetastore class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251979#comment-13251979 ] Carl Steinbach commented on HIVE-2928: -- @Mithun: That's great news! Thanks for experimenting with this. Is the patch ready for review? If so can you please click the submit patch button? Thanks. Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2947) Arc test -- IGNORE
Arc test -- IGNORE -- Key: HIVE-2947 URL: https://issues.apache.org/jira/browse/HIVE-2947 Project: Hive Issue Type: Bug Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-2928: --- Fix Version/s: 0.9.0 Status: Patch Available (was: Open) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Fix For: 0.9.0 Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Problems with Arc/Phabricator
Hi John,
Regarding the test plans: Carl, could you be more specific about what
is going wrong so I can attempt to reproduce the problem?
At some point Arc started requiring that the commit message contain a Test
Plan string, or maybe this has always been a requirement and it was just
automatically added before? Anyway, right now you have to manually add this
or you get the following error:
% git log -1
commit 2649ca167182bb02823b3fb00bbe7602f591717e
Author: Carl Steinbach c...@cloudera.com
Date: Wed Apr 11 15:12:53 2012 -0700
HIVE-2947. Test Phabricator
% arc diff --trace --jira HIVE-2947
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive-test/.arc_jira_lib'...
[0] conduit conduit.connect()
[0] conduit 329,414 us
[1] exec $ (cd '/Users/carl/Work/repos/hive-test'; git rev-parse
--show-cdup)
[1] exec 16,731 us
[2] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git rev-parse
--verify HEAD^)
[2] exec 20,879 us
[3] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git log
--first-parent --format=medium 'HEAD^'..HEAD)
[3] exec 17,852 us
[4] conduit differential.parsecommitmessage()
[4] conduit 558,248 us
Fatal error: Uncaught exception
'ArcanistDifferentialCommitMessageParserException' with message 'Invalid or
missing field 'Test Plan': You must provide a test plan.' in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
Stack trace:
#0
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
ArcanistDifferentialCommitMessage-pullDataFromConduit(Object(ConduitClient))
#1
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(368):
ArcJIRAConfiguration-willRunDiffWorkflow()
#2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
ArcJIRAConfiguration-willRunWorkflow('diff', Object(ArcanistDiffWorkflow))
#3 {main}
thrown in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
on line 88
Thanks.
Carl
Re: Problems with Arc/Phabricator
Hi John, For HIVE-2646: the diagnostics aren't great, but I think it's just choking on a patch that needs to be rebased, since as Kevin says, if I download the latest .patch from JIRA and apply that with patch -p0, I get the correct failure message. So just ask the author to rebase again? I updated my local arcanist and libphutil repos and then tried applying the patch for HIVE-2646 using the command 'arc patch --revision 2133'. This works. However, when I download the patch for HIVE-2646 and manually apply it, I get the following error: % wget https://issues.apache.org/jira/secure/attachment/12522177/HIVE-2646.D2133.14.patch % patch -p0 HIVE-2646.D2133.14.patch patching file build-common.xml patching file build.properties patching file build.xml patching file builtins/build.xml patching file builtins/ivy.xml patching file cli/ivy.xml patching file common/ivy.xml patching file contrib/build.xml patching file contrib/ivy.xml patching file hbase-handler/build.xml patching file hbase-handler/ivy.xml patching file hwi/build.xml patching file hwi/ivy.xml patching file ivy/common-configurations.xml patch: malformed patch at line 622: Index: ivy/ivysettings.xml Other people have complained about problems encountered while manually applying Phabricator diffs, and the response has always been that the patch needs to be rebased. However, I think this example indicates that there's actually some other problem here. Thanks. Carl
Re: Problems with Arc/Phabricator
+1 to switching over to Git. As for the rest of the Phabricator/Gerrit/Reviewboard discussion, I think we should pick this up again at the contributor meeting on Wednesday. Thanks. Carl On Wed, Apr 11, 2012 at 12:19 PM, Ashutosh Chauhan hashut...@apache.orgwrote: +1 on moving away from arc/phabricator. It works great when it works, but most of the time it doesnt work. Ashutosh On Wed, Apr 11, 2012 at 11:57, Owen O'Malley omal...@apache.org wrote: On Wed, Apr 11, 2012 at 11:48 AM, Edward Capriolo edlinuxg...@gmail.com wrote: If we are going to switch from fabricator we just might as well go back to not using anything. Review board was really clunky and confusing. I'm mostly +1 to that. If no one is supporting phabricator, then it won't work for long. Personally, I'd love it if we could move Hive to git completely. Has anyone used gerrit? The videos of it make it look better than sliced bread. -- Owen
Re: Problems with Arc/Phabricator
CC'ing David Recordon, who can probably help with a point of contact
for coordinating future Phabricator upgrades.
It looks like the test plan problem mentioned below (which affects
git, but not svn) was introduced when the reviews.facebook.net
Phabricator server was upgraded Feb 23. I've committed a change to
the arc-jira module which should deal with it:
https://github.com/facebook/arc-jira/commit/b62b5976ec9a974ed102c2f55b530edde48cfaa5
So if you run ant arc-setup in your Hive sandbox, you should be good to go.
JVS
On Wed, Apr 11, 2012 at 3:37 PM, Carl Steinbach c...@cloudera.com wrote:
Hi John,
Regarding the test plans: Carl, could you be more specific about what
is going wrong so I can attempt to reproduce the problem?
At some point Arc started requiring that the commit message contain a Test
Plan string, or maybe this has always been a requirement and it was just
automatically added before? Anyway, right now you have to manually add this
or you get the following error:
% git log -1
commit 2649ca167182bb02823b3fb00bbe7602f591717e
Author: Carl Steinbach c...@cloudera.com
Date: Wed Apr 11 15:12:53 2012 -0700
HIVE-2947. Test Phabricator
% arc diff --trace --jira HIVE-2947
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive-test/.arc_jira_lib'...
[0] conduit conduit.connect()
[0] conduit 329,414 us
[1] exec $ (cd '/Users/carl/Work/repos/hive-test'; git rev-parse
--show-cdup)
[1] exec 16,731 us
[2] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git rev-parse
--verify HEAD^)
[2] exec 20,879 us
[3] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git log
--first-parent --format=medium 'HEAD^'..HEAD)
[3] exec 17,852 us
[4] conduit differential.parsecommitmessage()
[4] conduit 558,248 us
Fatal error: Uncaught exception
'ArcanistDifferentialCommitMessageParserException' with message 'Invalid or
missing field 'Test Plan': You must provide a test plan.' in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
Stack trace:
#0
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
ArcanistDifferentialCommitMessage-pullDataFromConduit(Object(ConduitClient))
#1
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(368):
ArcJIRAConfiguration-willRunDiffWorkflow()
#2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
ArcJIRAConfiguration-willRunWorkflow('diff', Object(ArcanistDiffWorkflow))
#3 {main}
thrown in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
on line 88
Thanks.
Carl
Re: Problems with Arc/Phabricator
Is mac only supported OS ? Arc doesn't work for me on linux, which is
unfortunate since thats where I do all my testing.
Thanks,
Ashutosh
On Wed, Apr 11, 2012 at 17:37, John Sichi jsi...@gmail.com wrote:
CC'ing David Recordon, who can probably help with a point of contact
for coordinating future Phabricator upgrades.
It looks like the test plan problem mentioned below (which affects
git, but not svn) was introduced when the reviews.facebook.net
Phabricator server was upgraded Feb 23. I've committed a change to
the arc-jira module which should deal with it:
https://github.com/facebook/arc-jira/commit/b62b5976ec9a974ed102c2f55b530edde48cfaa5
So if you run ant arc-setup in your Hive sandbox, you should be good to go.
JVS
On Wed, Apr 11, 2012 at 3:37 PM, Carl Steinbach c...@cloudera.com wrote:
Hi John,
Regarding the test plans: Carl, could you be more specific about what
is going wrong so I can attempt to reproduce the problem?
At some point Arc started requiring that the commit message contain a
Test
Plan string, or maybe this has always been a requirement and it was just
automatically added before? Anyway, right now you have to manually add
this
or you get the following error:
% git log -1
commit 2649ca167182bb02823b3fb00bbe7602f591717e
Author: Carl Steinbach c...@cloudera.com
Date: Wed Apr 11 15:12:53 2012 -0700
HIVE-2947. Test Phabricator
% arc diff --trace --jira HIVE-2947
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive-test/.arc_jira_lib'...
[0] conduit conduit.connect()
[0] conduit 329,414 us
[1] exec $ (cd '/Users/carl/Work/repos/hive-test'; git rev-parse
--show-cdup)
[1] exec 16,731 us
[2] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git rev-parse
--verify HEAD^)
[2] exec 20,879 us
[3] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git log
--first-parent --format=medium 'HEAD^'..HEAD)
[3] exec 17,852 us
[4] conduit differential.parsecommitmessage()
[4] conduit 558,248 us
Fatal error: Uncaught exception
'ArcanistDifferentialCommitMessageParserException' with message 'Invalid
or
missing field 'Test Plan': You must provide a test plan.' in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
Stack trace:
#0
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
ArcanistDifferentialCommitMessage-pullDataFromConduit(Object(ConduitClient))
#1
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(368):
ArcJIRAConfiguration-willRunDiffWorkflow()
#2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
ArcJIRAConfiguration-willRunWorkflow('diff',
Object(ArcanistDiffWorkflow))
#3 {main}
thrown in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
on line 88
Thanks.
Carl
Re: Problems with Arc/Phabricator
I took a look at the 14.patch file in JIRA; the last hunk for ivy/common-configurations.xml ends in \ No newline at end of file, and I think that's what is screwing it up (since that immediately precedes ivysettings.xml). If I delete that line, the patch applies (although it causes a reject for common-configurations.xml). So perhaps there's a bug with arc's diff options used to produce the patch file in the first place...seems like that should be something easy to reproduce and track down (by editing the end of a file which currently is missing a trailing newline). JVS On Wed, Apr 11, 2012 at 4:11 PM, Carl Steinbach c...@cloudera.com wrote: Hi John, For HIVE-2646: the diagnostics aren't great, but I think it's just choking on a patch that needs to be rebased, since as Kevin says, if I download the latest .patch from JIRA and apply that with patch -p0, I get the correct failure message. So just ask the author to rebase again? I updated my local arcanist and libphutil repos and then tried applying the patch for HIVE-2646 using the command 'arc patch --revision 2133'. This works. However, when I download the patch for HIVE-2646 and manually apply it, I get the following error: % wget https://issues.apache.org/jira/secure/attachment/12522177/HIVE-2646.D2133.14.patch % patch -p0 HIVE-2646.D2133.14.patch patching file build-common.xml patching file build.properties patching file build.xml patching file builtins/build.xml patching file builtins/ivy.xml patching file cli/ivy.xml patching file common/ivy.xml patching file contrib/build.xml patching file contrib/ivy.xml patching file hbase-handler/build.xml patching file hbase-handler/ivy.xml patching file hwi/build.xml patching file hwi/ivy.xml patching file ivy/common-configurations.xml patch: malformed patch at line 622: Index: ivy/ivysettings.xml Other people have complained about problems encountered while manually applying Phabricator diffs, and the response has always been that the patch needs to be rebased. However, I think this example indicates that there's actually some other problem here. Thanks. Carl
Re: Problems with Arc/Phabricator
arc works fine on Linux for me. What problem are you encountering?
JVS
On Wed, Apr 11, 2012 at 5:45 PM, Ashutosh Chauhan
ashutosh.chau...@gmail.com wrote:
Is mac only supported OS ? Arc doesn't work for me on linux, which is
unfortunate since thats where I do all my testing.
Thanks,
Ashutosh
On Wed, Apr 11, 2012 at 17:37, John Sichi jsi...@gmail.com wrote:
CC'ing David Recordon, who can probably help with a point of contact
for coordinating future Phabricator upgrades.
It looks like the test plan problem mentioned below (which affects
git, but not svn) was introduced when the reviews.facebook.net
Phabricator server was upgraded Feb 23. I've committed a change to
the arc-jira module which should deal with it:
https://github.com/facebook/arc-jira/commit/b62b5976ec9a974ed102c2f55b530edde48cfaa5
So if you run ant arc-setup in your Hive sandbox, you should be good to
go.
JVS
On Wed, Apr 11, 2012 at 3:37 PM, Carl Steinbach c...@cloudera.com wrote:
Hi John,
Regarding the test plans: Carl, could you be more specific about what
is going wrong so I can attempt to reproduce the problem?
At some point Arc started requiring that the commit message contain a
Test
Plan string, or maybe this has always been a requirement and it was
just
automatically added before? Anyway, right now you have to manually add
this
or you get the following error:
% git log -1
commit 2649ca167182bb02823b3fb00bbe7602f591717e
Author: Carl Steinbach c...@cloudera.com
Date: Wed Apr 11 15:12:53 2012 -0700
HIVE-2947. Test Phabricator
% arc diff --trace --jira HIVE-2947
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive-test/.arc_jira_lib'...
[0] conduit conduit.connect()
[0] conduit 329,414 us
[1] exec $ (cd '/Users/carl/Work/repos/hive-test'; git rev-parse
--show-cdup)
[1] exec 16,731 us
[2] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git rev-parse
--verify HEAD^)
[2] exec 20,879 us
[3] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git log
--first-parent --format=medium 'HEAD^'..HEAD)
[3] exec 17,852 us
[4] conduit differential.parsecommitmessage()
[4] conduit 558,248 us
Fatal error: Uncaught exception
'ArcanistDifferentialCommitMessageParserException' with message 'Invalid
or
missing field 'Test Plan': You must provide a test plan.' in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
Stack trace:
#0
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
ArcanistDifferentialCommitMessage-pullDataFromConduit(Object(ConduitClient))
#1
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(368):
ArcJIRAConfiguration-willRunDiffWorkflow()
#2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
ArcJIRAConfiguration-willRunWorkflow('diff',
Object(ArcanistDiffWorkflow))
#3 {main}
thrown in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
on line 88
Thanks.
Carl
Re: Problems with Arc/Phabricator
Oops, actually adding d...@fb.com this time.
On Wed, Apr 11, 2012 at 5:37 PM, John Sichi jsi...@gmail.com wrote:
CC'ing David Recordon, who can probably help with a point of contact
for coordinating future Phabricator upgrades.
It looks like the test plan problem mentioned below (which affects
git, but not svn) was introduced when the reviews.facebook.net
Phabricator server was upgraded Feb 23. I've committed a change to
the arc-jira module which should deal with it:
https://github.com/facebook/arc-jira/commit/b62b5976ec9a974ed102c2f55b530edde48cfaa5
So if you run ant arc-setup in your Hive sandbox, you should be good to go.
JVS
On Wed, Apr 11, 2012 at 3:37 PM, Carl Steinbach c...@cloudera.com wrote:
Hi John,
Regarding the test plans: Carl, could you be more specific about what
is going wrong so I can attempt to reproduce the problem?
At some point Arc started requiring that the commit message contain a Test
Plan string, or maybe this has always been a requirement and it was just
automatically added before? Anyway, right now you have to manually add this
or you get the following error:
% git log -1
commit 2649ca167182bb02823b3fb00bbe7602f591717e
Author: Carl Steinbach c...@cloudera.com
Date: Wed Apr 11 15:12:53 2012 -0700
HIVE-2947. Test Phabricator
% arc diff --trace --jira HIVE-2947
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive-test/.arc_jira_lib'...
[0] conduit conduit.connect()
[0] conduit 329,414 us
[1] exec $ (cd '/Users/carl/Work/repos/hive-test'; git rev-parse
--show-cdup)
[1] exec 16,731 us
[2] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git rev-parse
--verify HEAD^)
[2] exec 20,879 us
[3] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git log
--first-parent --format=medium 'HEAD^'..HEAD)
[3] exec 17,852 us
[4] conduit differential.parsecommitmessage()
[4] conduit 558,248 us
Fatal error: Uncaught exception
'ArcanistDifferentialCommitMessageParserException' with message 'Invalid or
missing field 'Test Plan': You must provide a test plan.' in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
Stack trace:
#0
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
ArcanistDifferentialCommitMessage-pullDataFromConduit(Object(ConduitClient))
#1
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(368):
ArcJIRAConfiguration-willRunDiffWorkflow()
#2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
ArcJIRAConfiguration-willRunWorkflow('diff', Object(ArcanistDiffWorkflow))
#3 {main}
thrown in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
on line 88
Thanks.
Carl
[jira] [Created] (HIVE-2948) HiveFileFormatUtils should use Path.SEPARATOR instead of File.Separator
HiveFileFormatUtils should use Path.SEPARATOR instead of File.Separator --- Key: HIVE-2948 URL: https://issues.apache.org/jira/browse/HIVE-2948 Project: Hive Issue Type: Bug Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Because its munging hdfs paths and not OS paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2948) HiveFileFormatUtils should use Path.SEPARATOR instead of File.Separator
[ https://issues.apache.org/jira/browse/HIVE-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2948: -- Attachment: HIVE-2948.D2763.1.patch ashutoshc requested code review of HIVE-2948 [jira] HiveFileFormatUtils should use Path.SEPARATOR instead of File.Separator. Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2948 Simple patch which replaces File.separator to Path.SEPARATOR since separator for hdfs path is Path.SEPARATOR not java's File.SEPARATOR Because its munging hdfs paths and not OS paths. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2763 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6291/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. HiveFileFormatUtils should use Path.SEPARATOR instead of File.Separator --- Key: HIVE-2948 URL: https://issues.apache.org/jira/browse/HIVE-2948 Project: Hive Issue Type: Bug Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-2948.D2763.1.patch Because its munging hdfs paths and not OS paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Problems with Arc/Phabricator
I have arc working on linux. Fedora core 12 I think.
On Wed, Apr 11, 2012 at 8:45 PM, Ashutosh Chauhan
ashutosh.chau...@gmail.com wrote:
Is mac only supported OS ? Arc doesn't work for me on linux, which is
unfortunate since thats where I do all my testing.
Thanks,
Ashutosh
On Wed, Apr 11, 2012 at 17:37, John Sichi jsi...@gmail.com wrote:
CC'ing David Recordon, who can probably help with a point of contact
for coordinating future Phabricator upgrades.
It looks like the test plan problem mentioned below (which affects
git, but not svn) was introduced when the reviews.facebook.net
Phabricator server was upgraded Feb 23. I've committed a change to
the arc-jira module which should deal with it:
https://github.com/facebook/arc-jira/commit/b62b5976ec9a974ed102c2f55b530edde48cfaa5
So if you run ant arc-setup in your Hive sandbox, you should be good to go.
JVS
On Wed, Apr 11, 2012 at 3:37 PM, Carl Steinbach c...@cloudera.com wrote:
Hi John,
Regarding the test plans: Carl, could you be more specific about what
is going wrong so I can attempt to reproduce the problem?
At some point Arc started requiring that the commit message contain a
Test
Plan string, or maybe this has always been a requirement and it was just
automatically added before? Anyway, right now you have to manually add
this
or you get the following error:
% git log -1
commit 2649ca167182bb02823b3fb00bbe7602f591717e
Author: Carl Steinbach c...@cloudera.com
Date: Wed Apr 11 15:12:53 2012 -0700
HIVE-2947. Test Phabricator
% arc diff --trace --jira HIVE-2947
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive-test/.arc_jira_lib'...
[0] conduit conduit.connect()
[0] conduit 329,414 us
[1] exec $ (cd '/Users/carl/Work/repos/hive-test'; git rev-parse
--show-cdup)
[1] exec 16,731 us
[2] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git rev-parse
--verify HEAD^)
[2] exec 20,879 us
[3] exec $ (cd '/Users/carl/Work/repos/hive-test/'; git log
--first-parent --format=medium 'HEAD^'..HEAD)
[3] exec 17,852 us
[4] conduit differential.parsecommitmessage()
[4] conduit 558,248 us
Fatal error: Uncaught exception
'ArcanistDifferentialCommitMessageParserException' with message 'Invalid
or
missing field 'Test Plan': You must provide a test plan.' in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
Stack trace:
#0
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
ArcanistDifferentialCommitMessage-pullDataFromConduit(Object(ConduitClient))
#1
/Users/carl/Work/repos/hive-test/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(368):
ArcJIRAConfiguration-willRunDiffWorkflow()
#2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
ArcJIRAConfiguration-willRunWorkflow('diff',
Object(ArcanistDiffWorkflow))
#3 {main}
thrown in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
on line 88
Thanks.
Carl
Re: Problems with Arc/Phabricator
I think the most practical solution is try and use arc/phab and then if there is a problem fall back to Jira and do it the old way. Edward On Wed, Apr 11, 2012 at 7:17 PM, Carl Steinbach c...@cloudera.com wrote: +1 to switching over to Git. As for the rest of the Phabricator/Gerrit/Reviewboard discussion, I think we should pick this up again at the contributor meeting on Wednesday. Thanks. Carl On Wed, Apr 11, 2012 at 12:19 PM, Ashutosh Chauhan hashut...@apache.orgwrote: +1 on moving away from arc/phabricator. It works great when it works, but most of the time it doesnt work. Ashutosh On Wed, Apr 11, 2012 at 11:57, Owen O'Malley omal...@apache.org wrote: On Wed, Apr 11, 2012 at 11:48 AM, Edward Capriolo edlinuxg...@gmail.com wrote: If we are going to switch from fabricator we just might as well go back to not using anything. Review board was really clunky and confusing. I'm mostly +1 to that. If no one is supporting phabricator, then it won't work for long. Personally, I'd love it if we could move Hive to git completely. Has anyone used gerrit? The videos of it make it look better than sliced bread. -- Owen
[jira] [Commented] (HIVE-2888) Creating an external table using the 'LIKE' clause actually creates a managed table.
[ https://issues.apache.org/jira/browse/HIVE-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252183#comment-13252183 ] Kai Zhang commented on HIVE-2888: - This has been fixed in HIVE-2086. Creating an external table using the 'LIKE' clause actually creates a managed table. Key: HIVE-2888 URL: https://issues.apache.org/jira/browse/HIVE-2888 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.1 Reporter: Craig Swift Original Estimate: 12h Remaining Estimate: 12h When creating a new table with both the EXTERNAL and LIKE clauses, the new table does not behave as an EXTERNAL table thus resulting in potential data loss. Example: create external table test1 (VAL string) location '/user/craig/test1'; create external table test2 like test1 location '/user/craig/test2'; drop table test1; /user/craig/test1 - still exists drop table test2; /user/craig/test2 - is deleted (bad) If I do an extended describe on both tables, test1 shows up as being external while test2 is a managed table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2888) Creating an external table using the 'LIKE' clause actually creates a managed table.
[ https://issues.apache.org/jira/browse/HIVE-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-2888. Resolution: Fixed Fix Version/s: 0.8.0 Fixed in HIVE-2086. Creating an external table using the 'LIKE' clause actually creates a managed table. Key: HIVE-2888 URL: https://issues.apache.org/jira/browse/HIVE-2888 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.1 Reporter: Craig Swift Fix For: 0.8.0 Original Estimate: 12h Remaining Estimate: 12h When creating a new table with both the EXTERNAL and LIKE clauses, the new table does not behave as an EXTERNAL table thus resulting in potential data loss. Example: create external table test1 (VAL string) location '/user/craig/test1'; create external table test2 like test1 location '/user/craig/test2'; drop table test1; /user/craig/test1 - still exists drop table test2; /user/craig/test2 - is deleted (bad) If I do an extended describe on both tables, test1 shows up as being external while test2 is a managed table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
