Re: [ANNOUNCE] New Hive Committer - Yin Huai

2013-09-04 Thread yongqiang he
Congrats!

On Wed, Sep 4, 2013 at 10:23 AM, Jason Dere jd...@hortonworks.com wrote:
 Yin, congrats!

 Jason

 On Sep 4, 2013, at 7:54 AM, Eugene Koifman ekoif...@hortonworks.com wrote:

 Congrats!


 On Wed, Sep 4, 2013 at 5:23 AM, Brock Noland br...@cloudera.com wrote:

 Congrats Yin!!

 On Wed, Sep 4, 2013 at 4:14 AM, Lefty Leverenz leftylever...@gmail.com
 wrote:
 Bravo, Yin!

 -- Lefty


 On Wed, Sep 4, 2013 at 4:17 AM, Sushanth Sowmyan khorg...@gmail.com
 wrote:

 Congrats, Yin! :)
 On Sep 4, 2013 1:13 AM, Alexander Alten-Lorenz wget.n...@gmail.com
 wrote:

 Amazing news, congratz Yin! Well deserved!

 On Sep 4, 2013, at 6:49 AM, Carl Steinbach c...@apache.org wrote:

 The Apache Hive PMC has voted to make Yin Huai a committer on the
 Apache
 Hive project.

 Please join me in congratulating Yin!

 Thanks.

 Carl

 --
 Alexander Alten-Lorenz
 http://mapredit.blogspot.com
 German Hadoop LinkedIn Group: http://goo.gl/N8pCF






 --
 Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


Re: [VOTE] Apache Hive 0.11.0 Release Candidate 2

2013-05-15 Thread yongqiang he
+1.

build and test pass for me.



On Tue, May 14, 2013 at 7:06 PM, Thejas Nair the...@hortonworks.com wrote:

 +1 (non binding)

 - Verified md5 checksums for hive-0.11.0.tar.gz   and
 hive-0.11.0-bin.tar.gz
 - Verified gpg keys for  hive-0.11.0.tar.gz   and hive-0.11.0-bin.tar.gz
 - Built hive using the source release
 - Ran simple hive queries using new build , binary in   hive-0.11.0.tar.gz
   and hive-0.11.0-bin.tar.gz  in local mode
 - Ran simple hive queries against hadoop 1.x cluster using build from
 source release
 - Ran hcat cli and webhcat requests against hadoop 1.x cluster

 The hcat and webhcat scripts don't have the execute permissions. But since
 the vote is on the source release , my vote is +1 . I have created
 https://issues.apache.org/jira/browse/HIVE-4559 to track this issue.





 On Sat, May 11, 2013 at 10:33 AM, Owen O'Malley omal...@apache.org
 wrote:

  Based on feedback from everyone, I have respun release candidate, RC2.
  Please take a look. We've fixed 7 problems with the previous RC:
  * Release notes were incorrect
   * HIVE-4018 - MapJoin failing with Distributed Cache error
   * HIVE-4421 - Improve memory usage by ORC dictionaries
   * HIVE-4500 - Ensure that HiveServer 2 closes log files.
   * HIVE-4494 - ORC map columns get class cast exception in some contexts
   * HIVE-4498 - Fix TestBeeLineWithArgs failure
   * HIVE-4505 - Hive can't load transforms with remote scripts
   * HIVE-4527 - Fix the eclipse template
 
  Source tag for RC2 is at:
 
  https://svn.apache.org/repos/asf/hive/tags/release-0.11.0rc2
 
 
  Source tar ball and convenience binary artifacts can be found
  at: http://people.apache.org/~omalley/hive-0.11.0rc2/
 
  This release has many goodies including HiveServer2, integrated
  hcatalog, windowing and analytical functions, decimal data type,
  better query planning, performance enhancements and various bug fixes.
  In total, we resolved more than 350 issues. Full list of fixed issues
  can be found at:  http://s.apache.org/8Fr
 
 
  Voting will conclude in 72 hours.
 
  Hive PMC Members: Please test and vote.
 
  Thanks,
 
  Owen
 



Re: [VOTE] Amend Hive Bylaws + Add HCatalog Submodule

2013-01-28 Thread yongqiang he
+1 and +1

On Mon, Jan 28, 2013 at 3:09 PM, Owen O'Malley omal...@apache.org wrote:
 +1 and +1


 On Mon, Jan 28, 2013 at 1:56 PM, Ashish Thusoo athu...@qubole.com wrote:

 Measure 1: +1
 Measure 2: +1

 Ashish


 On Mon, Jan 28, 2013 at 1:11 PM, Ashutosh Chauhan hashut...@apache.org
 wrote:

  Measure 1: +1
  Measure 2: +1
 
  Ashutosh
 
 
  On Mon, Jan 28, 2013 at 11:48 AM, Carl Steinbach c...@apache.org wrote:
 
  Measure 1: +1 (binding)
  Measure 2: +1 (binding)
 
  On Mon, Jan 28, 2013 at 11:47 AM, Carl Steinbach c...@apache.org
 wrote:
 
   I am calling a vote on the following two measures.
  
   Measure 1: Amend Hive Bylaws to Define Submodules and Submodule
  Committers
  
   If this measure passes the Apache Hive Project Bylaws will be
   amended with the following changes:
  
  
  
 
 https://cwiki.apache.org/confluence/display/Hive/Proposed+Changes+to+Hive+Bylaws+for+Submodule+Committers
  
   The motivation for these changes is discussed in the following
   email thread which appeared on the hive-dev and hcatalog-dev
   mailing lists:
  
   http://markmail.org/thread/u5nap7ghvyo7euqa
  
  
   Measure 2: Create HCatalog Submodule and Adopt HCatalog Codebase
  
   This measure provides for 1) the establishment of an HCatalog
   submodule in the Apache Hive Project, 2) the adoption of the
   Apache HCatalog codebase into the Hive HCatalog submodule, and
   3) adding all currently active HCatalog committers as submodule
   committers on the Hive HCatalog submodule.
  
   Passage of this measure depends on the passage of Measure 1.
  
  
   Voting:
  
   Both measures require +1 votes from 2/3 of active Hive PMC
   members in order to pass. All participants in the Hive project
   are encouraged to vote on these measures, but only votes from
   active Hive PMC members are binding. The voting period
   commences immediately and shall last a minimum of six days.
  
   Voting is carried out by replying to this email thread. You must
   indicate which measure you are voting on in order for your vote
   to be counted.
  
   More details about the voting process can be found in the Apache
   Hive Project Bylaws:
  
   https://cwiki.apache.org/confluence/display/Hive/Bylaws
  
  
 
 
 



multi-cluster support in hive

2012-01-17 Thread yongqiang he
Hi hive-dev,

We are planning to make hive run across multiple data centers
(physical clusters). We prefer to use hive metastore to provide a
unified namespace.
Tables/partitions can exist in more than one cluster. And one cluster
is defined as a primary cluster. A primary cluster is a table level
property. A table T1's primary cluster is C1 meaning :1) C1 contains
all data that is available in all other clusters. 2) write is only
allowed in this cluster for table C1. but need to allow exceptions
here 3) new partitions are only allowed to be created in C1.  4) all
data changes to T1 happened in the primary cluster should be
replicated to other clusters if there are any secondary clusters. but
there should be a conf to disable it as there are some exception
situations.

The first thing that needs to be done is to make hive metastore have a
concept of cluster. And that also means all thrift communication calls
to metastore need to provide a cluster parameter. So we have there
options here:
1) add a cluster parameter to existing thrift interfaces
or
2) add new interfaces which do exactly the same set of functionalities
as old ones but using a different name (use _on_cluster suffifx
maybe?) and have a cluster parameter
or
3) overwrite database name for the purpose of cluster name. And allow
a table co-exist in multiple databases. But that require to promote
table to top level citizen, and degrade database. For example, show
tables used to scan all tables in current db, but now need to scan
all tables in all databases.

We would like to get more ideas about which one to choose, and we are
definitely open to other alternatives that we missed here.

We are also looking for other systems that have solved similar
problems. If anyone knows such a system, we would like to know.
Appreciate that!

This is tracked on jira https://issues.apache.org/jira/browse/HIVE-2612.

Thanks
Yongqiang


multi-cluster support in hive

2012-01-17 Thread yongqiang he
Hi hive-dev,

We are planning to make hive run across multiple data centers
(physical clusters). We prefer to use hive metastore to provide a
unified namespace.
Tables/partitions can exist in more than one cluster. And one cluster
is defined as a primary cluster. A primary cluster is a table level
property. A table T1's primary cluster is C1 meaning :1) C1 contains
all data that is available in all other clusters. 2) write is only
allowed in this cluster for table C1. but need to allow exceptions
here 3) new partitions are only allowed to be created in C1.  4) all
data changes to T1 happened in the primary cluster should be
replicated to other clusters if there are any secondary clusters. but
there should be a conf to disable it as there are some exception
situations.

The first thing that needs to be done is to make hive metastore have a
concept of cluster. And that also means all thrift communication calls
to metastore need to provide a cluster parameter. So we have there
options here:
1) add a cluster parameter to existing thrift interfaces
or
2) add new interfaces which do exactly the same set of functionalities
as old ones but using a different name (use _on_cluster suffifx
maybe?) and have a cluster parameter
or
3) overwrite database name for the purpose of cluster name. And allow
a table co-exist in multiple databases. But that require to promote
table to top level citizen, and degrade database. For example, show
tables used to scan all tables in current db, but now need to scan
all tables in all databases.

We would like to get more ideas about which one to choose, and we are
definitely open to other alternatives that we missed here.

We are also looking for other systems that have solved similar
problems. If anyone knows such a system, we would like to know.
Appreciate that!

This is tracked on jira https://issues.apache.org/jira/browse/HIVE-2612.

Thanks


Re: Hive 0.8.1 RC0 on Friday

2012-01-17 Thread yongqiang he
https://issues.apache.org/jira/browse/HIVE-2718 should be included.

Thanks
Yongqiang
On Tue, Jan 17, 2012 at 3:00 PM, Carl Steinbach c...@cloudera.com wrote:
 Hi,

 I plan to start cutting release candidates for Hive 0.8.1 on Friday (1/20).
 The primary driver for 0.8.1 is the Apache Bigtop project which needs a
 Hive 0.8 release artifact by the end of the month that includes HIVE-2631
 and HIVE-2629. Please email me in advance if you want to backport any other
 patches to 0.8.1.

 Thanks.

 Carl


Re: Review Request: Support archiving for multiple partitions if the table is partitioned by multiple columns

2011-10-30 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2595/
---

(Updated 2011-10-30 07:34:36.866186)


Review request for hive and namit jain.


Changes
---

Address review comments except the last one:
Why have a for loop - you are returning inside if anyway the first time ?

I think it is safe to return even for the first time. For any partition 
returned, it should either be archived or not. If the archive is based on its 
parent partition col, the partition should also be marked as archived, 
otherwise, it should not be there.


Summary
---

Support archiving for multiple partitions if the table is partitioned by 
multiple columns


This addresses bug HIVE-2278.
https://issues.apache.org/jira/browse/HIVE-2278


Diffs (updated)
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1190490 
  trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1190490 
  
trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 1190490 
  trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1190490 
  trunk/data/files/archive_corrupt.rc UNKNOWN 
  trunk/metastore/if/hive_metastore.thrift 1190490 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1190490 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1190490 
  
trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java
 1190490 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java 
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1190490 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1190490 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 1190490 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 
1190490 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1190490 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1190490 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
1195116 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
1190490 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
1190490 
  trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec4.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec5.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive1.q.out 1190490 
  trunk/ql/src/test/results/clientnegative/archive2.q.out 1190490 
  trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION 
  

Review Request: Support archiving for multiple partitions if the table is partitioned by multiple columns

2011-10-28 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2595/
---

Review request for hive and namit jain.


Summary
---

Support archiving for multiple partitions if the table is partitioned by 
multiple columns


This addresses bug HIVE-2278.
https://issues.apache.org/jira/browse/HIVE-2278


Diffs
-

  trunk/data/files/archive_corrupt.rc UNKNOWN 
  trunk/metastore/if/hive_metastore.thrift 1188906 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1188906 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1188906 
  
trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java
 1188906 
  
trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 1188906 
  trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1188906 
  trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java 
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1188906 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 
1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
1188906 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
1188906 
  trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive1.q.out 1188906 
  trunk/ql/src/test/results/clientnegative/archive2.q.out 1188906 
  trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/2595/diff


Testing
---

Support archiving for multiple partitions if the table is partitioned by 
multiple columns


Thanks,

Yongqiang



Re: Review Request: HIVE-2484: Improve ALTER TABLE SET SERDE to work on partition level

2011-10-05 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2215/#review2362
---



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
https://reviews.apache.org/r/2215/#comment5422

code format here.




trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
https://reviews.apache.org/r/2215/#comment5424

dup this line and move them to if (part!=null){} and else{}


- Yongqiang


On 2011-10-05 18:40:55, xiaol wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/2215/
 ---
 
 (Updated 2011-10-05 18:40:55)
 
 
 Review request for hive and Yongqiang He.
 
 
 Summary
 ---
 
 Allowed ALTER TABLE SET SERDE and ALTER TABLE SET SERDEPROPERTIES to accept 
 the PARTITION qualifier to work on a specific partition.
 
 
 This addresses bug HIVE-2484.
 https://issues.apache.org/jira/browse/HIVE-2484
 
 
 Diffs
 -
 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1178944 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 1178944 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1178944 
   
 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
  1178944 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1178984 
   trunk/ql/src/test/queries/clientpositive/alter_table_serde.q PRE-CREATION 
   trunk/ql/src/test/results/clientpositive/alter_table_serde.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/2215/diff
 
 
 Testing
 ---
 
 
 Thanks,
 
 xiaol
 




Re: Review Request: Log more Hadoop task counter values in the MapRedStats object.

2011-10-04 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2167/#review2316
---



trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java
https://reviews.apache.org/r/2167/#comment5322

don't put the counter names here, let's use a map and pass it the hook


- Yongqiang


On 2011-10-04 17:27:43, Kevin Wilfong wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/2167/
 ---
 
 (Updated 2011-10-04 17:27:43)
 
 
 Review request for hive, Ramkumar Vadali and Yongqiang He.
 
 
 Summary
 ---
 
 I added the counters mentioned in the task to the MapRedStats class, and 
 modified HadoopJobExecHelper to collect them.
 
 I got tired of writing the same code over and over again, so I modified the 
 way MapRedStats and HadoopJobExecHelper treat task counters.  MapRedStats now 
 has an enum with all of the task counters we want to collect, it is a subset 
 of the enum in Task$Counter.  Task is package private so the enum in it is 
 unavailable.  MapRedStats now contains a map from the enum values to the 
 values of the counters, if they were set.  HadoopJobExecHelper loops over the 
 enum values and tries to get a value for each counter.  As long as the new 
 getter and setter methods are used the functionality is the same, in 
 particular for the getter, if a counter was set, it returns the value of the 
 counter, otherwise it returns -1.
 
 
 This addresses bug Hive-2479.
 https://issues.apache.org/jira/browse/Hive-2479
 
 
 Diffs
 -
 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 1178612 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 
 1178612 
 
 Diff: https://reviews.apache.org/r/2167/diff
 
 
 Testing
 ---
 
 I ran some queries to verify the counters were being populated.
 
 I also ran a few of the unit test queries to verify I hadn't broken anything.
 
 
 Thanks,
 
 Kevin
 




Re: Review Request: Log more Hadoop task counter values in the MapRedStats object.

2011-10-04 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2167/#review2335
---



trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java
https://reviews.apache.org/r/2167/#comment5396

remove the enum, just either put all the counters objects into this 
MapRedStats, or put a mapcounterGrpAndName, counterInst here.

For simplicity, we can just put the counters object here and hooks can 
anything they want.


- Yongqiang


On 2011-10-04 22:58:24, Kevin Wilfong wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/2167/
 ---
 
 (Updated 2011-10-04 22:58:24)
 
 
 Review request for hive, Ramkumar Vadali and Yongqiang He.
 
 
 Summary
 ---
 
 I added the counters mentioned in the task to the MapRedStats class, and 
 modified HadoopJobExecHelper to collect them.
 
 I got tired of writing the same code over and over again, so I modified the 
 way MapRedStats and HadoopJobExecHelper treat task counters.  MapRedStats now 
 has an enum with all of the task counters we want to collect, it is a subset 
 of the enum in Task$Counter.  Task is package private so the enum in it is 
 unavailable.  MapRedStats now contains a map from the enum values to the 
 values of the counters, if they were set.  HadoopJobExecHelper loops over the 
 enum values and tries to get a value for each counter.  As long as the new 
 getter and setter methods are used the functionality is the same, in 
 particular for the getter, if a counter was set, it returns the value of the 
 counter, otherwise it returns -1.
 
 
 This addresses bug Hive-2479.
 https://issues.apache.org/jira/browse/Hive-2479
 
 
 Diffs
 -
 
   trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1178612 
   trunk/conf/hive-default.xml 1178612 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 1178612 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 
 1178612 
 
 Diff: https://reviews.apache.org/r/2167/diff
 
 
 Testing
 ---
 
 I ran some queries to verify the counters were being populated.
 
 I also ran a few of the unit test queries to verify I hadn't broken anything.
 
 
 Thanks,
 
 Kevin
 




Re: Review Request: move lock retry logic into ZooKeeperHiveLockManager

2011-09-27 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
---

(Updated 2011-09-27 21:11:09.644951)


Review request for hive and Ning Zhang.


Changes
---

fix the failing testcase and more logging


Summary
---

move lock retry logic into ZooKeeperHiveLockManager


This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450


Diffs (updated)
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255 
  trunk/conf/hive-default.xml 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java 
1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 
1171255 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 1171255 
  trunk/ql/src/test/queries/clientnegative/lockneg1.q 1171255 
  trunk/ql/src/test/queries/clientnegative/lockneg2.q 1171255 
  trunk/ql/src/test/queries/clientnegative/lockneg3.q 1171255 
  trunk/ql/src/test/queries/clientnegative/lockneg4.q 1171255 

Diff: https://reviews.apache.org/r/1950/diff


Testing
---

will run tests locally


Thanks,

Yongqiang



Re: Review Request: move lock retry logic into ZooKeeperHiveLockManager

2011-09-26 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
---

(Updated 2011-09-26 17:56:19.501365)


Review request for hive and Ning Zhang.


Changes
---

address comments


Summary
---

move lock retry logic into ZooKeeperHiveLockManager


This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450


Diffs (updated)
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255 
  trunk/conf/hive-default.xml 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 
1171255 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 1171255 

Diff: https://reviews.apache.org/r/1950/diff


Testing
---

will run tests locally


Thanks,

Yongqiang



Re: Review Request: move lock retry logic into ZooKeeperHiveLockManager

2011-09-23 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
---

(Updated 2011-09-23 21:10:26.466632)


Review request for hive and Ning Zhang.


Changes
---

update comment in hive-default.xml


Summary
---

move lock retry logic into ZooKeeperHiveLockManager


This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450


Diffs (updated)
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255 
  trunk/conf/hive-default.xml 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 
1171255 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 1171255 

Diff: https://reviews.apache.org/r/1950/diff


Testing
---

will run tests locally


Thanks,

Yongqiang



Re: Review Request: Add method to PerfLogger to perform cleanup/final steps.

2011-09-22 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2023/#review2030
---



trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
https://reviews.apache.org/r/2023/#comment4579

one nitpick is that do you think call this function a 'close' will be more 
clear? we already have a PerfLogEnd().


- Yongqiang


On 2011-09-22 23:54:37, Kevin Wilfong wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/2023/
 ---
 
 (Updated 2011-09-22 23:54:37)
 
 
 Review request for hive, Yongqiang He and Ning Zhang.
 
 
 Summary
 ---
 
 I added a method called finishPerfLogger, which is called at the end of the 
 run method in Driver to allow PerfLogger subclasses to perform any cleanup 
 tasks or logging that requires all the perf stats to have been calculated.  
 It takes as parameters a Log, to be consistent with the other methods in 
 PerfLogger, as well as the QueryPlan from Driver in order to allow 
 information such as the query ID to be logged.
 
 I should note that finishPerfLogger is not guaranteed to run.  This was my 
 intention.
 
 
 This addresses bug HIVE-2461.
 https://issues.apache.org/jira/browse/HIVE-2461
 
 
 Diffs
 -
 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1173881 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1173881 
 
 Diff: https://reviews.apache.org/r/2023/diff
 
 
 Testing
 ---
 
 I created a subclass of PerfLogger which implemented the finishPerfLogger 
 method, and verified it was successfully called and executed.
 
 I also ran some of the unit tests to verify nothing was broken.
 
 
 Thanks,
 
 Kevin
 




Review Request: move lock retry logic into ZooKeeperHiveLockManager

2011-09-18 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
---

Review request for hive and Ning Zhang.


Summary
---

move lock retry logic into ZooKeeperHiveLockManager


This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450


Diffs
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255 
  trunk/conf/hive-default.xml 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 
1171255 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 1171255 

Diff: https://reviews.apache.org/r/1950/diff


Testing
---

will run tests locally


Thanks,

Yongqiang



Review Request: make hive mapper initialize faster when having tons of input files

2011-09-09 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1783/
---

Review request for hive and Ning Zhang.


Summary
---

when one hive job has tons of input files, a lot of mappers may fail because of 
slow initialization.


This addresses bug HIVE-2440.
https://issues.apache.org/jira/browse/HIVE-2440


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1167289 

Diff: https://reviews.apache.org/r/1783/diff


Testing
---


Thanks,

Yongqiang



Re: Review Request: get_privilege does not get user level privilege and should return emptylist instead of null

2011-09-06 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1646/#review1774
---



trunk/ql/src/test/queries/clientpositive/authorization_7.q
https://reviews.apache.org/r/1646/#comment4051

yes. what we want to do is to test the global privilege. The first command 
is to grant a global privilege, and after that do test with that. 


- Yongqiang


On 2011-08-24 22:23:39, Yongqiang He wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/1646/
 ---
 
 (Updated 2011-08-24 22:23:39)
 
 
 Review request for hive and Ning Zhang.
 
 
 Summary
 ---
 
 get_privilege does not get user level privilege and should return emptylist 
 instead of null
 
 
 This addresses bug HIVE-2405.
 https://issues.apache.org/jira/browse/HIVE-2405
 
 
 Diffs
 -
 
   
 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
 1161232 
   trunk/ql/src/test/queries/clientpositive/authorization_7.q PRE-CREATION 
   trunk/ql/src/test/results/clientpositive/authorization_7.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/1646/diff
 
 
 Testing
 ---
 
 
 Thanks,
 
 Yongqiang
 




Review Request: move validation of renaming/replacing columns from runtime to parser

2011-08-28 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1672/
---

Review request for hive and Ning Zhang.


Summary
---

move validation of renaming/replacing columns from runtime to parser


This addresses bug HIVE-2415.
https://issues.apache.org/jira/browse/HIVE-2415


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1162190 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1162190 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
1162190 
  trunk/ql/src/test/queries/clientnegative/replace_columns.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/replace_columns_2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/replace_columns_3.q PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/altern1.q.out 1162190 
  trunk/ql/src/test/results/clientnegative/column_rename1.q.out 1162190 
  trunk/ql/src/test/results/clientnegative/column_rename2.q.out 1162190 
  trunk/ql/src/test/results/clientnegative/column_rename4.q.out 1162190 
  trunk/ql/src/test/results/clientnegative/replace_columns.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/replace_columns_2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/replace_columns_3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/replace_columns_4.q.out PRE-CREATION 
  trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1162190 

Diff: https://reviews.apache.org/r/1672/diff


Testing
---


Thanks,

Yongqiang



Review Request: get_privilege does not get user level privilege

2011-08-24 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1644/
---

Review request for hive.


Summary
---

get_privilege does not get user level privilege


This addresses bug HIVE-2405.
https://issues.apache.org/jira/browse/HIVE-2405


Diffs
-

  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
1161232 
  trunk/ql/src/test/queries/clientpositive/authorization_7.q PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/authorization_7.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/1644/diff


Testing
---

added a new testcase


Thanks,

Yongqiang



Re: Review Request: Cli: Print Hadoop's CPU milliseconds

2011-06-24 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/948/#review904
---



trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
https://reviews.apache.org/r/948/#comment1947

why do you need to pull this var out of the code block? 



trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
https://reviews.apache.org/r/948/#comment1948

can you not store the MapRedStatsList in DriverContext?



trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
https://reviews.apache.org/r/948/#comment1949

format the code here



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
https://reviews.apache.org/r/948/#comment1959

how do you know the hadoop support this counter



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
https://reviews.apache.org/r/948/#comment1960

are u going to print this info if this job failed?



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
https://reviews.apache.org/r/948/#comment1958

remove 'ms'


- Yongqiang


On 2011-06-23 07:41:33, Siying Dong wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/948/
 ---
 
 (Updated 2011-06-23 07:41:33)
 
 
 Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
 
 
 Summary
 ---
 
 In hive CLI, print out CPU msec from Hadoop MapReduce coutners.
 
 
 This addresses bug HIVE-2236.
 https://issues.apache.org/jira/browse/HIVE-2236
 
 
 Diffs
 -
 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1138748 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1138748 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1138748 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 
 1138748 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1138748 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
 1138748 
 
 Diff: https://reviews.apache.org/r/948/diff
 
 
 Testing
 ---
 
 run the updated codes against real clusters and make sure it printing is 
 correct.
 
 
 Thanks,
 
 Siying
 




Re: trunk busted?

2011-06-22 Thread yongqiang he
database.q failed me when testing HIVE-2100

On Wed, Jun 22, 2011 at 2:23 PM, John Sichi jsi...@fb.com wrote:
 Yeah, that's one of the failures (out of many different ones) that Jenkins 
 has been hitting (see the end of this log):

 https://builds.apache.org/view/G-L/view/Hive/job/Hive-trunk-h0.21/788/console

 It's sporadic, probably based on server load.

 JVS

 On Jun 22, 2011, at 2:10 PM, Ning Zhang wrote:

 John, here's what I got for 'ant clean package'. It seems ivy is flaky now?


 ivy-download:
      [get] Getting: 
 http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
      [get] To: 
 /data/users/nzhang/reviews/2/apache-hive/build/ivy/lib/ivy-2.1.0.jar
      [get] Error getting 
 http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar to 
 /data/users/nzhang/reviews/2/apache-hive/build/ivy/lib/ivy-2.1.0.jar

 BUILD FAILED
 /data/users/nzhang/reviews/2/apache-hive/build.xml:196: The following error 
 occurred while executing this line:
 /data/users/nzhang/reviews/2/apache-hive/build.xml:130: The following error 
 occurred while executing this line:
 /data/users/nzhang/reviews/2/apache-hive/build-common.xml:128: 
 java.net.ConnectException: Connection refused


 On Jun 22, 2011, at 12:43 PM, John Sichi wrote:

 Yeah, all tests passed when I committed the bitmap indexes, so I'm not sure 
 what's up.

 JVS

 On Jun 22, 2011, at 12:36 PM, Ning Zhang wrote:

 trunk was fine the last time I committed. John the last ones who committed 
 were Carl (branching 0.7.1) and you (bitmap index). :) Did you get all the 
 tests passed? I'll test with a clean checkout.

 On Jun 22, 2011, at 12:02 PM, John Sichi wrote:

 Are other committers able to pass tests on Hive trunk?  I'm getting lots 
 of failures, and Jenkins seems to have been barfing for a while too.

 JVS








Re: Review Request: speedup addInputPaths

2011-06-15 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/898/
---

(Updated 2011-06-15 20:24:15.631412)


Review request for hive.


Changes
---

address Ning's comments. Did the minimum change and the performance is 
acceptable. We can try to remove empty path check if in future we see the 
latency is not good.


Summary
---

speedup addInputPaths


This addresses bug HIVE-2218.
https://issues.apache.org/jira/browse/HIVE-2218


Diffs (updated)
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1135335 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
1135335 

Diff: https://reviews.apache.org/r/898/diff


Testing
---

yes.


Thanks,

Yongqiang



Review Request: speedup addInputPaths

2011-06-14 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/898/
---

Review request for hive.


Summary
---

speedup addInputPaths


This addresses bug HIVE-2218.
https://issues.apache.org/jira/browse/HIVE-2218


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1135335 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1135335 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
1135335 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1135335 

Diff: https://reviews.apache.org/r/898/diff


Testing
---

yes.


Thanks,

Yongqiang



Review Request: HIVE-2100 virtual column references inside subqueries cause execution exceptions

2011-06-02 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/844/
---

Review request for hive.


Summary
---

HIVE-2100
virtual column references inside subqueries cause execution exceptions


This addresses bug HIVE-2100.
https://issues.apache.org/jira/browse/HIVE-2100


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1097824 
  trunk/ql/src/test/queries/clientpositive/nestedvirtual.q PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/nestedvirtual.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/844/diff


Testing
---


Thanks,

Yongqiang



Review Request: add exception handling to hive's record reader

2011-05-31 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/812/
---

Review request for hive.


Summary
---

add exception handling to hive's record reader


Diffs
-

  trunk/conf/hive-default.xml 1127229 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/BucketizedHiveRecordReader.java 
1101261 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1101261 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveRecordReader.java 1101261 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java 
1101261 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java
 1101261 
  trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 
1101261 
  trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 
1101261 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionHandler.java
 PRE-CREATION 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionHandlerChain.java
 PRE-CREATION 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionNextHandleResult.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/812/diff


Testing
---


Thanks,

Yongqiang



Re: Review Request: add exception handling to hive's record reader

2011-05-31 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/812/
---

(Updated 2011-05-31 20:31:23.581581)


Review request for hive.


Changes
---

add exception handling to hive's record reader


Summary
---

add exception handling to hive's record reader


This addresses bug HIVE-2154.
https://issues.apache.org/jira/browse/HIVE-2154


Diffs
-

  trunk/conf/hive-default.xml 1127229 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/BucketizedHiveRecordReader.java 
1101261 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1101261 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveRecordReader.java 1101261 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java 
1101261 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java
 1101261 
  trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 
1101261 
  trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 
1101261 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionHandler.java
 PRE-CREATION 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionHandlerChain.java
 PRE-CREATION 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionNextHandleResult.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/812/diff


Testing
---


Thanks,

Yongqiang



Re: Review Request: add exception handling to hive's record reader

2011-05-31 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/812/
---

(Updated 2011-06-01 00:38:42.863958)


Review request for hive.


Changes
---

address Ning's comments to move some code to hadoopshims


Summary
---

add exception handling to hive's record reader


This addresses bug HIVE-2154.
https://issues.apache.org/jira/browse/HIVE-2154


Diffs (updated)
-

  trunk/conf/hive-default.xml 1129953 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/BucketizedHiveRecordReader.java 
1129953 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1129953 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveRecordReader.java 1129953 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java 
1129953 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java
 1129953 
  trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 
1129953 
  trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 
1129953 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionHandler.java
 PRE-CREATION 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionHandlerChain.java
 PRE-CREATION 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionHandlerUtil.java
 PRE-CREATION 
  
trunk/shims/src/common/java/org/apache/hadoop/hive/io/HiveIOExceptionNextHandleResult.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/812/diff


Testing
---


Thanks,

Yongqiang



Review Request: Mappers across multiple partitions with different SerDe failed

2011-05-31 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/813/
---

Review request for hive.


Summary
---

Mappers across multiple partitions with different SerDe failed


This addresses bug HIVE-2187.
https://issues.apache.org/jira/browse/HIVE-2187


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
1129953 
  trunk/ql/src/test/queries/clientpositive/combine_check_serde.q PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/combine_check_serde.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/813/diff


Testing
---


Thanks,

Yongqiang



Re: Review Request: Too many open files in running negative cli tests

2011-05-09 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/696/
---

(Updated 2011-05-09 17:15:15.221624)


Review request for hive.


Changes
---

reverted the change in ZooKeeperHiveLockManager's close, but put a zookeeper 
close in removeReduantNodes
added more comments in Driver.


Summary
---

Too many open files in running negative cli tests


This addresses bug HIVE-2151.
https://issues.apache.org/jira/browse/HIVE-2151


Diffs (updated)
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1101116 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 1101116 

Diff: https://reviews.apache.org/r/696/diff


Testing
---


Thanks,

Yongqiang



Review Request: Too many open files in running negative cli tests

2011-05-06 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/696/
---

Review request for hive.


Summary
---

Too many open files in running negative cli tests


This addresses bug HIVE-2151.
https://issues.apache.org/jira/browse/HIVE-2151


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1100117 

Diff: https://reviews.apache.org/r/696/diff


Testing
---


Thanks,

Yongqiang



Re: Review Request: Too many open files in running negative cli tests

2011-05-06 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/696/
---

(Updated 2011-05-06 19:17:48.260294)


Review request for hive.


Changes
---

new diff


Summary
---

Too many open files in running negative cli tests


This addresses bug HIVE-2151.
https://issues.apache.org/jira/browse/HIVE-2151


Diffs (updated)
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1100117 

Diff: https://reviews.apache.org/r/696/diff


Testing
---


Thanks,

Yongqiang



Re: Review Request: alter table concatenate fails and deletes data

2011-04-26 Thread Yongqiang He


 On 2011-04-26 05:33:19, Carl Steinbach wrote:
  trunk/ql/src/test/queries/clientpositive/alter_merge_index.q, line 1
  https://reviews.apache.org/r/665/diff/1/?file=17323#file17323line1
 
  Please add some comments here too.

Carl, what kind of comments do you want here? 


 On 2011-04-26 05:33:19, Carl Steinbach wrote:
  trunk/ql/src/test/queries/clientnegative/alter_merge_index.q, line 1
  https://reviews.apache.org/r/665/diff/1/?file=17322#file17322line1
 
  Please add some comments explaining what it is that you're testing, 
  e.g. why is this a negative test? Why is expected to fail?

why is this a negative test? Why is expected to fail?
should i put a positive test on a negative test dir? Should we explain all test 
cases in client negative dir?
Come on, please put some positive suggestions.


- Yongqiang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/665/#review557
---


On 2011-04-25 22:47:17, Yongqiang He wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/665/
 ---
 
 (Updated 2011-04-25 22:47:17)
 
 
 Review request for hive.
 
 
 Summary
 ---
 
 alter table concatenate fails and deletes data
 
 It is because the number of reducers is set to -1.
 
 In this patch, it is set to zero. 
 
 Also added a move task as the child task of the merge task. added a conf to 
 control whether to check index or not, and add the job name for the merge job.
 
 
 This addresses bug HIVE-2125.
 https://issues.apache.org/jira/browse/HIVE-2125
 
 
 Diffs
 -
 
   trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 
   trunk/conf/hive-default.xml 1096599 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 
   
 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
  1096599 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 1096599 
   trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION 
   trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION 
   trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out 
 PRE-CREATION 
   trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/665/diff
 
 
 Testing
 ---
 
 
 Thanks,
 
 Yongqiang
 




Re: Review Request: alter table concatenate fails and deletes data

2011-04-26 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/665/
---

(Updated 2011-04-26 22:27:30.586885)


Review request for hive.


Changes
---

address Carl's comments about .q files


Summary
---

alter table concatenate fails and deletes data

It is because the number of reducers is set to -1.

In this patch, it is set to zero. 

Also added a move task as the child task of the merge task. added a conf to 
control whether to check index or not, and add the job name for the merge job.


This addresses bug HIVE-2125.
https://issues.apache.org/jira/browse/HIVE-2125


Diffs (updated)
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096925 
  trunk/conf/hive-default.xml 1096925 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096925 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096925 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 
1096925 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
1096925 
  trunk/ql/src/test/queries/clientnegative/alter_concatenate_indexed_table.q 
PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/alter_concatenate_indexed_table.q 
PRE-CREATION 
  
trunk/ql/src/test/results/clientnegative/alter_concatenate_indexed_table.q.out 
PRE-CREATION 
  
trunk/ql/src/test/results/clientpositive/alter_concatenate_indexed_table.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/665/diff


Testing
---


Thanks,

Yongqiang



Review Request: Hive's symlink text input format should be able to work with ComineHiveInputFormat

2011-04-25 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/653/
---

Review request for hive.


Summary
---

Hive's symlink text input format should be able to work with 
ComineHiveInputFormat


This addresses bug hive-2126.
https://issues.apache.org/jira/browse/hive-2126


Diffs
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096093 
  trunk/conf/hive-default.xml 1096093 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1096093 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096093 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/ReworkMapredInputFormat.java 
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java 
1096093 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
1096093 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
1096093 
  
trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 
1096093 

Diff: https://reviews.apache.org/r/653/diff


Testing
---


Thanks,

Yongqiang



Review Request: alter table concatenate fails and deletes data

2011-04-25 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/665/
---

Review request for hive.


Summary
---

alter table concatenate fails and deletes data

It is because the number of reducers is set to -1.

In this patch, it is set to zero. 

Also added a move task as the child task of the merge task. added a conf to 
control whether to check index or not, and add the job name for the merge job.


This addresses bug HIVE-2125.
https://issues.apache.org/jira/browse/HIVE-2125


Diffs
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 
  trunk/conf/hive-default.xml 1096599 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 
1096599 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
1096599 
  trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/665/diff


Testing
---


Thanks,

Yongqiang



Re: Review Request: alter table concatenate fails and deletes data

2011-04-25 Thread Yongqiang He


 On 2011-04-25 23:20:59, Ning Zhang wrote:
  trunk/conf/hive-default.xml, line 1042
  https://reviews.apache.org/r/665/diff/1/?file=17317#file17317line1042
 
  can you make the indentation consistent with the other property 
  elements?

It shows the same indentation on my local. So it might just be an review board 
display issue.


 On 2011-04-25 23:20:59, Ning Zhang wrote:
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java, 
  line 1203
  https://reviews.apache.org/r/665/diff/1/?file=17321#file17321line1203
 
  So after adding this, does the block-level merge after INSERT OVERWRITE 
  be automatically supported?

No. Not automatically supported. We still need to do some work there. but it is 
a separate issue.


- Yongqiang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/665/#review554
---


On 2011-04-25 22:47:17, Yongqiang He wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/665/
 ---
 
 (Updated 2011-04-25 22:47:17)
 
 
 Review request for hive.
 
 
 Summary
 ---
 
 alter table concatenate fails and deletes data
 
 It is because the number of reducers is set to -1.
 
 In this patch, it is set to zero. 
 
 Also added a move task as the child task of the merge task. added a conf to 
 control whether to check index or not, and add the job name for the merge job.
 
 
 This addresses bug HIVE-2125.
 https://issues.apache.org/jira/browse/HIVE-2125
 
 
 Diffs
 -
 
   trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 
   trunk/conf/hive-default.xml 1096599 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 
   
 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
  1096599 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 1096599 
   trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION 
   trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION 
   trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out 
 PRE-CREATION 
   trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/665/diff
 
 
 Testing
 ---
 
 
 Thanks,
 
 Yongqiang
 




sorry for the spam emails sent out by editing jira issue.

2011-04-20 Thread yongqiang he
Hi,

I just made some interesting jira tickets to be unassigned, so other
people can feel free to pick them up.

sorry again for the spams sent out by editing jira issue.


thanks
yongqiang


Re: sorry for the spam emails sent out by editing jira issue.

2011-04-20 Thread yongqiang he
Good to kown. Thanks for the info. Will use it next time

On Wednesday, April 20, 2011, Jean-Daniel Cryans jdcry...@apache.org wrote:
 Jira has a bulk change functionality that prevents spamming people
 like that. Once you get a list of jiras, there's a drop down called
 Tools and that's where you'll find it.

 J-D

 On Wed, Apr 20, 2011 at 5:10 PM, yongqiang he heyongqiang...@gmail.com 
 wrote:
 Hi,

 I just made some interesting jira tickets to be unassigned, so other
 people can feel free to pick them up.

 sorry again for the spams sent out by editing jira issue.


 thanks
 yongqiang




Review Request: No lock for some non-mapred tasks.

2011-04-18 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/617/
---

Review request for hive.


Summary
---

No lock for some non-mapred tasks.


This addresses bug HIVE-2109.
https://issues.apache.org/jira/browse/HIVE-2109


Diffs
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1094342 
  trunk/conf/hive-default.xml 1094342 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1094342 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1094342 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1094342 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java 1094342 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1094342 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 
1094342 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
1094342 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1094342 

Diff: https://reviews.apache.org/r/617/diff


Testing
---


Thanks,

Yongqiang



Re: [ANNOUNCE] New Hive Committer - Siying Dong

2011-04-14 Thread yongqiang he
Congrats Siying!

On Thu, Apr 14, 2011 at 7:29 AM, Edward Capriolo edlinuxg...@gmail.com wrote:
 Congrats Siying. Great work so far.

 On Thu, Apr 14, 2011 at 2:30 AM, Namit Jain nj...@fb.com wrote:
 Hi Folks,

 The Hive PMC has passed the vote to make Siying Dong a
 new committer on the Apache Hive project.

 Following is a list of the contributions that Siying has made to the project:

 http://bit.ly/goFnzw
 http://bit.ly/goFnzw
 Congratulations Siying.
 Please send over your CLA to Apache.


 Thanks,
 Namit




Review Request: auto map join bug

2011-04-07 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/559/
---

Review request for hive.


Summary
---

auto map join bug


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1088810 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
1088810 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java
 1088810 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java
 1088810 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java
 1088810 
  trunk/ql/src/test/queries/clientpositive/auto_join28.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/auto_join29.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/auto_join30.q PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/auto_join12.q.out 1088810 
  trunk/ql/src/test/results/clientpositive/auto_join20.q.out 1088810 
  trunk/ql/src/test/results/clientpositive/auto_join21.q.out 1088810 
  trunk/ql/src/test/results/clientpositive/auto_join28.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/auto_join29.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/auto_join3.q.out 1088810 
  trunk/ql/src/test/results/clientpositive/auto_join30.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/559/diff


Testing
---

yes.


Thanks,

Yongqiang



Re: Meanings of privileges

2011-03-22 Thread yongqiang he
INDEX - my best guess is that this allows me to create/drop indexes on a
table?
Yes. It is there for this purpose.

 Is it the case that if I have select access on a table, I can use
any index that exists on a table?
No. index is also a table now, so you need to have access to both of them.

LOCK - Presumably this allows users to lock or unlock a table, so maybe a
better question is: are these locks like mutexes, where only I can access
the table, or is this literally locking down the table, so it can't be
modified in any way?

Yes. If only you have lock privilege on this table, and concurrency is
enabled, no one will be able to run anything against the table.

SHOW_DATABASE - I'm not sure what the scope of this one is: if I don't have
show_database access, can I not use the show database command?

if you don't have show_database access, you should not be able to use
the show database command. I do not think today this privilege is
supported.

 create access on a table doesn't seem to have a lot of semantic value
i think create on a table means create partition

Similarly, I'm having a hard time rationalizing why I can grant SHOW_DATABASE 
on a table.
This should be a bug. Basically each privilege has its set of scope,
(can apply to db level or table level or column or user level,
non-exclusive)

Thanks
Yongqiang
On Tue, Mar 22, 2011 at 6:30 PM, Jonathan Natkins na...@cloudera.com wrote:
 Hi all,

 I'm trying to understand the meaning of some of the privileges in the
 system, and I'm a bit stumped on what some of them actually do.

 Privileges that confuse me:
 INDEX - my best guess is that this allows me to create/drop indexes on a
 table?  Is it the case that if I have select access on a table, I can use
 any index that exists on a table?
 LOCK - Presumably this allows users to lock or unlock a table, so maybe a
 better question is: are these locks like mutexes, where only I can access
 the table, or is this literally locking down the table, so it can't be
 modified in any way?
 SHOW_DATABASE - I'm not sure what the scope of this one is: if I don't have
 show_database access, can I not use the show database command? Or does this
 extend to not being able to see the tables within a database?

 It seems like you can grant some privileges on objects that don't have a lot
 of meaning, i.e. create access on a table doesn't seem to have a lot of
 semantic value, unless Hive requires that permission to create indexes on a
 table, or something along those lines.  Similarly, I'm having a hard time
 rationalizing why I can grant SHOW_DATABASE on a table.

 Thanks a lot,
 Jon



Review Request: HIVE-1950

2011-02-03 Thread Yongqiang He

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/388/
---

Review request for hive.


Summary
---

early review


This addresses bug HIVE-1950.
https://issues.apache.org/jira/browse/HIVE-1950


Diffs
-

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java 1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHook.java 
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Throttle.java 1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java 1067036 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/MergeWork.java 
PRE-CREATION 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java
 PRE-CREATION 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeOutputFormat.java
 PRE-CREATION 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeRecordReader.java
 PRE-CREATION 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileKeyBufferWrapper.java
 PRE-CREATION 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java
 PRE-CREATION 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileValueBufferWrapper.java
 PRE-CREATION 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AlterTablePartMergeFilesDesc.java
 PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1067036 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1067036 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1067036 
  trunk/ql/src/test/queries/clientpositive/alter_merge.q PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/alter_merge.q.out PRE-CREATION 
  trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 
1067036 
  trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 
1067036 
  trunk/shims/src/common/java/org/apache/hadoop/hive/shims/CombineHiveKey.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/388/diff


Testing
---


Thanks,

Yongqiang



Re: patch review process

2011-01-20 Thread yongqiang he
 also think excluding small patches
 from
  this requirement is a bad idea because rational people can disagree about
  what qualifies as a small patch and what does not, and I'd like people to
  make ReviewBoard a habit instead of something that they use occasionally.
 I
  think that Yongqiang's point about scaring away new contributors with
 lots
  of requirements is valid, and I'm more that willing to post a review
 request
  for a first (or second) time contributor, but in general it's important
 for
  the contributor to create the request since only the creator can update
 it.
 
  Thanks.
 
  Carl
 
 
 
 
 
  On Wed, Jan 19, 2011 at 6:48 PM, yongqiang he heyongqiang...@gmail.com
 wrote:
 
  +1 for option 2.
 
  In general, we as a community should be nice to all contributors, and
  should avoid doing things that make contributors not comfortable, even
  that requires some work from committers. Sometimes it is especially
  true for new contributors, like we need to be more patience for new
  people. It seems a free style and contribution focused environment
  would be better to encourage people to do more contributions of
  different kinds.
 
  thanks
  -yongqiang
  On Wed, Jan 19, 2011 at 6:37 PM, Namit Jain nj...@fb.com wrote:
 
 
 
  It would be good to have a policy for submitting a new patch for
 review.
  If the patch is small, usually it is pretty easy to review.But, if it
  large,
  a GUI like reviewboard (https://reviews.apache.org) makes it easy.
 
  So, going forward, I would like to propose either of the following.
 
  1. All patches must go through reviewboard
  2. If a contributor/reviewer creates a reviewboard request,
     all subsequent review requests should go through the reviewboard.
 
 
  I would personally vote for 2., since for small patches, we don’t
 really
  need a
  reviewboard.
 
  But, please vote, and based on that, we can come up with a policy.
  Let us know, if you think of some other option.
 
  Thanks,
  -namit
 
 
 




 --
 Todd Lipcon
 Software Engineer, Cloudera



Re: patch review process

2011-01-19 Thread yongqiang he
+1 for option 2.

In general, we as a community should be nice to all contributors, and
should avoid doing things that make contributors not comfortable, even
that requires some work from committers. Sometimes it is especially
true for new contributors, like we need to be more patience for new
people. It seems a free style and contribution focused environment
would be better to encourage people to do more contributions of
different kinds.

thanks
-yongqiang
On Wed, Jan 19, 2011 at 6:37 PM, Namit Jain nj...@fb.com wrote:



 It would be good to have a policy for submitting a new patch for review.
 If the patch is small, usually it is pretty easy to review.But, if it large,
 a GUI like reviewboard (https://reviews.apache.org) makes it easy.

 So, going forward, I would like to propose either of the following.

 1. All patches must go through reviewboard
 2. If a contributor/reviewer creates a reviewboard request,
     all subsequent review requests should go through the reviewboard.


 I would personally vote for 2., since for small patches, we don’t really need 
 a
 reviewboard.

 But, please vote, and based on that, we can come up with a policy.
 Let us know, if you think of some other option.

 Thanks,
 -namit




Re: [ANNOUNCE] New Hive Committer - Amareshwari Sriramadasu

2010-11-09 Thread Yongqiang He
Congrats amareshwari!

Yongqiang

On Nov 8, 2010, at 6:00 PM, Namit Jain nj...@facebook.com wrote:

 Hi Folks,
 
 The Hive PMC has passed the vote to make Amareshwari Sriramadasu a
 new committer on the Apache Hive project.
 
 Following is a list of the contributions that Amareshwari has made to the
 project:
 
 http://bit.ly/c3z0ty
 
 Congratulations Amareshwari.
 Please send over your CLA to Apache.
 
 
 Thanks,
 Namit


Re: [ANNOUNCE] New Committer - Carl Steinbach

2010-11-02 Thread yongqiang he
Congrats Carl.

On Tue, Nov 2, 2010 at 11:27 AM, Edward Capriolo edlinuxg...@gmail.com wrote:
 On Tue, Nov 2, 2010 at 2:23 PM, Ashish Thusoo athu...@facebook.com wrote:
 Hi Folks,

 The Hive PMC has passed the vote to make Carl Steinbach a new committer on 
 the Apache Hive project. Carl has made a lot of contributions to Hive with 
 the latest being him serving as the release manager for 0.6.0 release. 
 Following is a list of some of the contributions that he has made to the 
 project:

 http://bit.ly/bu5rHq

 Congratulations Carl!! Please send over your CLA to Apache.

 Thanks,
 Ashish



 Carl,

 Congrats. Nice to have you aboard.

 Edward