[jira] [Commented] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it

2013-04-29 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645175#comment-13645175
 ] 

Namit Jain commented on HIVE-4209:
--

Thanks [~navis]

Looks good. Can you commit it if tests pass ?

+1

 Cache evaluation result of deterministic expression and reuse it
 

 Key: HIVE-4209
 URL: https://issues.apache.org/jira/browse/HIVE-4209
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-4209.6.patch.txt, HIVE-4209.D9585.1.patch, 
 HIVE-4209.D9585.2.patch, HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch, 
 HIVE-4209.D9585.5.patch


 For example, 
 {noformat}
 select key from src where key + 1  100 AND key + 1  200 limit 3;
 {noformat}
 key + 1 need not to be evaluated twice.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4440) SMB Operator spills to disk like it's 1999

2013-04-29 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645178#comment-13645178
 ] 

Namit Jain commented on HIVE-4440:
--

I really like the title of the jira.

Changing the parameter name is backward incompatible.
Can you support both the current parameter and the proposed parameter for now ?
Document it clearly, and say that the current parameter 
hive.mapjoin.bucket.cache.size will not be supported
for this from 0.13 or something like that.

 SMB Operator spills to disk like it's 1999
 --

 Key: HIVE-4440
 URL: https://issues.apache.org/jira/browse/HIVE-4440
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
 Attachments: HIVE-4440.1.patch


 I was recently looking into some performance issue with a query that used SMB 
 join and was running really slow. Turns out that the SMB join by default 
 caches only 100 values per key before spilling to disk. That seems overly 
 conservative to me. Changing the parameter resulted in a ~5x speedup - quite 
 significant.
 The parameter is: hive.mapjoin.bucket.cache.size
 Which right now is only used the SMB Operator as far as I can tell.
 The parameter was introduced originally (3 yrs ago) for the map join operator 
 (looks like pre-SMB) and set to 100 to avoid OOM. That seems to have been in 
 a different context though where you had to avoid running out of memory with 
 the cached hash table in the same process, I think.
 Two things I'd like to propose:
 a) Rename it to what it does: hive.smbjoin.cache.rows
 b) Set it to something less restrictive: 1
 If you string together a 5 table smb join with a map join and a map-side 
 group by aggregation you might still run out of memory, but the renamed 
 parameter should be easier to find and reduce. For most queries, I would 
 think that 1 is still a reasonable number to cache (On the reduce side we 
 use 25000 for shuffle joins).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4409) Prevent incompatible column type changes

2013-04-26 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642699#comment-13642699
 ] 

Namit Jain commented on HIVE-4409:
--

my bad, will do

 Prevent incompatible column type changes
 

 Key: HIVE-4409
 URL: https://issues.apache.org/jira/browse/HIVE-4409
 Project: Hive
  Issue Type: Improvement
  Components: CLI, Metastore
Affects Versions: 0.10.0
Reporter: Dilip Joseph
Assignee: Dilip Joseph
Priority: Minor
 Fix For: 0.12.0

 Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, 
 HIVE-4409.D10539.2.patch


 If a user changes the type of an existing column of a partitioned table to an 
 incompatible type, subsequent accesses of old partitions will result in a 
 ClassCastException (see example below).  We should prevent the user from 
 making incompatible type changes.  This feature will be controlled by a new 
 config parameter.
 Example:
 CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds 
 STRING) STORED AS SEQUENCEFILE;
 INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, 
 b1) FROM src LIMIT 1;
 SELECT * from test_table123 WHERE ds=foo1;
 SET hive.metastore.disallow.invalid.col.type.changes=true;
 ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
 SELECT * from test_table123 WHERE ds=foo1;
 The last SELECT fails with the following exception:
 Failed with exception java.io.IOException:java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
 java.io.IOException: java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
   at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
   at 
 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409

2013-04-26 Thread Namit Jain (JIRA) 
Namit Jain created HIVE-4424:


 Summary: MetaStoreUtils.java.orig checked in mistakenly by 
HIVE-4409
 Key: HIVE-4424
 URL: https://issues.apache.org/jira/browse/HIVE-4424
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Fix For: 0.12.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409

2013-04-26 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642700#comment-13642700
 ] 

Namit Jain commented on HIVE-4424:
--

Deleted 
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig

 MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
 ---

 Key: HIVE-4424
 URL: https://issues.apache.org/jira/browse/HIVE-4424
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Fix For: 0.12.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409

2013-04-26 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4424:
-

Status: Patch Available  (was: Open)

 MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
 ---

 Key: HIVE-4424
 URL: https://issues.apache.org/jira/browse/HIVE-4424
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Fix For: 0.12.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409

2013-04-26 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4424:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
 ---

 Key: HIVE-4424
 URL: https://issues.apache.org/jira/browse/HIVE-4424
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Fix For: 0.12.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4409) Prevent incompatible column type changes

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4409:
-

Status: Open  (was: Patch Available)

comments

 Prevent incompatible column type changes
 

 Key: HIVE-4409
 URL: https://issues.apache.org/jira/browse/HIVE-4409
 Project: Hive
  Issue Type: Improvement
  Components: CLI, Metastore
Affects Versions: 0.10.0
Reporter: Dilip Joseph
Assignee: Dilip Joseph
Priority: Minor
 Attachments: HIVE-4409.D10539.1.patch


 If a user changes the type of an existing column of a partitioned table to an 
 incompatible type, subsequent accesses of old partitions will result in a 
 ClassCastException (see example below).  We should prevent the user from 
 making incompatible type changes.  This feature will be controlled by a new 
 config parameter.
 Example:
 CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds 
 STRING) STORED AS SEQUENCEFILE;
 INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, 
 b1) FROM src LIMIT 1;
 SELECT * from test_table123 WHERE ds=foo1;
 SET hive.metastore.disallow.invalid.col.type.changes=true;
 ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
 SELECT * from test_table123 WHERE ds=foo1;
 The last SELECT fails with the following exception:
 Failed with exception java.io.IOException:java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
 java.io.IOException: java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
   at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
   at 
 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3952) merge map-job followed by map-reduce job

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3952:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Vinod

 merge map-job followed by map-reduce job
 

 Key: HIVE-3952
 URL: https://issues.apache.org/jira/browse/HIVE-3952
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Namit Jain
Assignee: Vinod Kumar Vavilapalli
 Fix For: 0.12.0

 Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, 
 HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, 
 HIVE-3952-20130424.txt


 Consider the query like:
 select count(*) FROM
 ( select idOne, idTwo, value FROM
   bigTable   
   JOIN
 
   smallTableOne on (bigTable.idOne = smallTableOne.idOne) 
   
   ) firstjoin 
 
 JOIN  
 
 smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
 where smallTableOne and smallTableTwo are smaller than 
 hive.auto.convert.join.noconditionaltask.size and
 hive.auto.convert.join.noconditionaltask is set to true.
 The joins are collapsed into mapjoins, and it leads to a map-only job
 (for the map-joins) followed by a map-reduce job (for the group by).
 Ideally, the map-only job should be merged with the following map-reduce job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4005) Column truncation

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4005:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Kevin

 Column truncation
 -

 Key: HIVE-4005
 URL: https://issues.apache.org/jira/browse/HIVE-4005
 Project: Hive
  Issue Type: New Feature
  Components: CLI
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Fix For: 0.12.0

 Attachments: HIVE-4005.1.patch.txt, HIVE-4005.2.patch.txt, 
 HIVE-4005.3.patch.txt, HIVE-4005.4.patch.txt, HIVE-4005.5.patch.txt, 
 HIVE-4005.6.patch.txt, HIVE-4005.6.patch.txt, HIVE-4005.7.patch.txt


 Column truncation allows users to remove data for columns that are no longer 
 useful.
 This is done by removing the data for the column and setting the length of 
 the column data and related lengths to 0 in the RC file header.
 RC file was fixed to recognize columns with lengths of zero to be empty and 
 are treated as if the column doesn't exist in the data, a null is returned 
 for every value of that column in every row. This is the same thing that 
 happens when more columns are selected than exist in the file.
 A new command was added to the CLI
 TRUNCATE TABLE ... PARTITION ... COLUMNS ...
 This launches a map only job where each mapper rewrites a single file without 
 the unnecessary column data and the adjusted headers. It does not 
 uncompress/deserialize the data so it is much faster than rewriting the data 
 with NULLs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4409) Prevent incompatible column type changes

2013-04-25 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641552#comment-13641552
 ] 

Namit Jain commented on HIVE-4409:
--

+1

 Prevent incompatible column type changes
 

 Key: HIVE-4409
 URL: https://issues.apache.org/jira/browse/HIVE-4409
 Project: Hive
  Issue Type: Improvement
  Components: CLI, Metastore
Affects Versions: 0.10.0
Reporter: Dilip Joseph
Assignee: Dilip Joseph
Priority: Minor
 Attachments: HIVE-4409.D10539.1.patch, HIVE-4409.D10539.2.patch


 If a user changes the type of an existing column of a partitioned table to an 
 incompatible type, subsequent accesses of old partitions will result in a 
 ClassCastException (see example below).  We should prevent the user from 
 making incompatible type changes.  This feature will be controlled by a new 
 config parameter.
 Example:
 CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds 
 STRING) STORED AS SEQUENCEFILE;
 INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, 
 b1) FROM src LIMIT 1;
 SELECT * from test_table123 WHERE ds=foo1;
 SET hive.metastore.disallow.invalid.col.type.changes=true;
 ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
 SELECT * from test_table123 WHERE ds=foo1;
 The last SELECT fails with the following exception:
 Failed with exception java.io.IOException:java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
 java.io.IOException: java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
   at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
   at 
 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it

2013-04-25 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641556#comment-13641556
 ] 

Namit Jain commented on HIVE-4209:
--

Can you refresh one more time - sorry, the parallel test framework is not 
applying the patch cleanly.

?   conf/hive-default.xml.template.orig
M   conf/hive-default.xml.template
?   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
M   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java
A  +ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeNullEvaluator.java
A  +ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java
M   
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java
M   
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java
M   ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
M   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java



Can you rebase the patch so that there no +, the patch does not apply cleanly 
in that case.

 Cache evaluation result of deterministic expression and reuse it
 

 Key: HIVE-4209
 URL: https://issues.apache.org/jira/browse/HIVE-4209
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-4209.D9585.1.patch, HIVE-4209.D9585.2.patch, 
 HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch, HIVE-4209.D9585.5.patch


 For example, 
 {noformat}
 select key from src where key + 1  100 AND key + 1  200 limit 3;
 {noformat}
 key + 1 need not to be evaluated twice.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4209:
-

Status: Open  (was: Patch Available)

 Cache evaluation result of deterministic expression and reuse it
 

 Key: HIVE-4209
 URL: https://issues.apache.org/jira/browse/HIVE-4209
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-4209.D9585.1.patch, HIVE-4209.D9585.2.patch, 
 HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch, HIVE-4209.D9585.5.patch


 For example, 
 {noformat}
 select key from src where key + 1  100 AND key + 1  200 limit 3;
 {noformat}
 key + 1 need not to be evaluated twice.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3781) not all meta events call metastore event listener

2013-04-25 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641560#comment-13641560
 ] 

Namit Jain commented on HIVE-3781:
--

This patch is also not applying cleanly.

~/hive/hive2$ svn stat
M   metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java
M   
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
A  +
metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterIndexEvent.java
A  +
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreDropIndexEvent.java
A  +
metastore/src/java/org/apache/hadoop/hive/metastore/events/AddIndexEvent.java
A  +
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAlterIndexEvent.java
A  +
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAddIndexEvent.java
M   
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreEventContext.java
A  +
metastore/src/java/org/apache/hadoop/hive/metastore/events/DropIndexEvent.java
M   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
M   
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java
M   ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
M   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java
M   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
?   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java.orig
M   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

 not all meta events call metastore event listener
 -

 Key: HIVE-3781
 URL: https://issues.apache.org/jira/browse/HIVE-3781
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.9.0
Reporter: Sudhanshu Arora
Assignee: Navis
 Attachments: hive.3781.3.patch, hive.3781.4.patch, 
 HIVE-3781.D7731.1.patch, HIVE-3781.D7731.2.patch, HIVE-3781.D7731.3.patch, 
 HIVE-3781.D7731.4.patch


 An event listener must be called for any DDL activity. For example, 
 create_index, drop_index today does not call metaevent listener. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3781) not all meta events call metastore event listener

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3781:
-

Status: Open  (was: Patch Available)

 not all meta events call metastore event listener
 -

 Key: HIVE-3781
 URL: https://issues.apache.org/jira/browse/HIVE-3781
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.9.0
Reporter: Sudhanshu Arora
Assignee: Navis
 Attachments: hive.3781.3.patch, hive.3781.4.patch, 
 HIVE-3781.D7731.1.patch, HIVE-3781.D7731.2.patch, HIVE-3781.D7731.3.patch, 
 HIVE-3781.D7731.4.patch


 An event listener must be called for any DDL activity. For example, 
 create_index, drop_index today does not call metaevent listener. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4409) Prevent incompatible column type changes

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4409:
-

Attachment: hive.4409.1.patch

 Prevent incompatible column type changes
 

 Key: HIVE-4409
 URL: https://issues.apache.org/jira/browse/HIVE-4409
 Project: Hive
  Issue Type: Improvement
  Components: CLI, Metastore
Affects Versions: 0.10.0
Reporter: Dilip Joseph
Assignee: Dilip Joseph
Priority: Minor
 Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, 
 HIVE-4409.D10539.2.patch


 If a user changes the type of an existing column of a partitioned table to an 
 incompatible type, subsequent accesses of old partitions will result in a 
 ClassCastException (see example below).  We should prevent the user from 
 making incompatible type changes.  This feature will be controlled by a new 
 config parameter.
 Example:
 CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds 
 STRING) STORED AS SEQUENCEFILE;
 INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, 
 b1) FROM src LIMIT 1;
 SELECT * from test_table123 WHERE ds=foo1;
 SET hive.metastore.disallow.invalid.col.type.changes=true;
 ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
 SELECT * from test_table123 WHERE ds=foo1;
 The last SELECT fails with the following exception:
 Failed with exception java.io.IOException:java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
 java.io.IOException: java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
   at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
   at 
 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4095) Add exchange partition in Hive

2013-04-25 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641594#comment-13641594
 ] 

Namit Jain commented on HIVE-4095:
--

The parallel test failed - refreshing/running again

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, 
 hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4095) Add exchange partition in Hive

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4095:
-

Attachment: hive.4095.refresh.patch

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, 
 hive.4095.refresh.patch, hive.4095.svn.thrift.patch, 
 hive.4095.svn.thrift.patch.refresh




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly

2013-04-25 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641603#comment-13641603
 ] 

Namit Jain commented on HIVE-4068:
--

+1

[~navis], can you commit it if tests pass

 Size of aggregation buffer which uses non-primitive type is not estimated 
 correctly
 ---

 Key: HIVE-4068
 URL: https://issues.apache.org/jira/browse/HIVE-4068
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, 
 HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch


 Currently, hive assumes an aggregation buffer which holds a map is occupying 
 just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can 
 be thrown (especially for 1k buffer). 
 workaround : set hive.map.aggr.hash.percentmemory=smaller value than 
 default(0.5)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4095) Add exchange partition in Hive

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4095:
-

Status: Patch Available  (was: Open)

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, 
 hive.4095.refresh.patch, hive.4095.svn.thrift.patch, 
 hive.4095.svn.thrift.patch.refresh




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4095) Add exchange partition in Hive

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4095:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Dheeraj

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Fix For: 0.12.0

 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, 
 hive.4095.refresh.patch, hive.4095.svn.thrift.patch, 
 hive.4095.svn.thrift.patch.refresh




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4340) ORC should provide raw data size

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4340:
-

Attachment: hive.4340.1.patch

 ORC should provide raw data size
 

 Key: HIVE-4340
 URL: https://issues.apache.org/jira/browse/HIVE-4340
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4340.1.patch.txt, HIVE-4340.2.patch.txt


 ORC's SerDe currently does nothing, and hence does not calculate a raw data 
 size.  WriterImpl, however, has enough information to provide one.
 WriterImpl should compute a raw data size for each row, aggregate them per 
 stripe and record it in the strip information, as RC currently does in its 
 key header, and allow the FileSinkOperator access to the size per row.
 FileSinkOperator should be able to get the raw data size from either the 
 SerDe or the RecordWriter when the RecordWriter can provide it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4340) ORC should provide raw data size

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4340:
-

Attachment: (was: hive.4340.1.patch)

 ORC should provide raw data size
 

 Key: HIVE-4340
 URL: https://issues.apache.org/jira/browse/HIVE-4340
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4340.1.patch.txt, HIVE-4340.2.patch.txt


 ORC's SerDe currently does nothing, and hence does not calculate a raw data 
 size.  WriterImpl, however, has enough information to provide one.
 WriterImpl should compute a raw data size for each row, aggregate them per 
 stripe and record it in the strip information, as RC currently does in its 
 key header, and allow the FileSinkOperator access to the size per row.
 FileSinkOperator should be able to get the raw data size from either the 
 SerDe or the RecordWriter when the RecordWriter can provide it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4340) ORC should provide raw data size

2013-04-25 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641758#comment-13641758
 ] 

Namit Jain commented on HIVE-4340:
--

Compilation is failing:

[javac] 
/Users/njain/hive/hive_commit3/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java:986:
 abstract method write(java.lang.Object) in 
org.apache.hadoop.hive.ql.io.orc.WriterImpl.TreeWriter cannot be accessed 
directly
[javac]   super.write(obj);
[javac]^

 ORC should provide raw data size
 

 Key: HIVE-4340
 URL: https://issues.apache.org/jira/browse/HIVE-4340
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4340.1.patch.txt, HIVE-4340.2.patch.txt


 ORC's SerDe currently does nothing, and hence does not calculate a raw data 
 size.  WriterImpl, however, has enough information to provide one.
 WriterImpl should compute a raw data size for each row, aggregate them per 
 stripe and record it in the strip information, as RC currently does in its 
 key header, and allow the FileSinkOperator access to the size per row.
 FileSinkOperator should be able to get the raw data size from either the 
 SerDe or the RecordWriter when the RecordWriter can provide it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4409) Prevent incompatible column type changes

2013-04-25 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4409:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Dilip

 Prevent incompatible column type changes
 

 Key: HIVE-4409
 URL: https://issues.apache.org/jira/browse/HIVE-4409
 Project: Hive
  Issue Type: Improvement
  Components: CLI, Metastore
Affects Versions: 0.10.0
Reporter: Dilip Joseph
Assignee: Dilip Joseph
Priority: Minor
 Fix For: 0.12.0

 Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, 
 HIVE-4409.D10539.2.patch


 If a user changes the type of an existing column of a partitioned table to an 
 incompatible type, subsequent accesses of old partitions will result in a 
 ClassCastException (see example below).  We should prevent the user from 
 making incompatible type changes.  This feature will be controlled by a new 
 config parameter.
 Example:
 CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds 
 STRING) STORED AS SEQUENCEFILE;
 INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, 
 b1) FROM src LIMIT 1;
 SELECT * from test_table123 WHERE ds=foo1;
 SET hive.metastore.disallow.invalid.col.type.changes=true;
 ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
 SELECT * from test_table123 WHERE ds=foo1;
 The last SELECT fails with the following exception:
 Failed with exception java.io.IOException:java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
 java.io.IOException: java.lang.ClassCastException: 
 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
 cannot be cast to 
 org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
   at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
   at 
 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4300:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Roshan

 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Fix For: 0.12.0

 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
 # modified:   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote
 #
 # Untracked files:
 #   (use git add file... to include in what will be committed)
 #
 # serde/src/gen/thrift/gen-cpp/complex_constants.cpp
 # serde/src/gen/thrift/gen-cpp/complex_constants.h
 # serde/src/gen/thrift/gen-cpp/complex_types.cpp
 # serde/src/gen/thrift/gen-cpp/complex_types.h
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp
 # serde/src/gen/thrift/gen-cpp

[jira] [Created] (HIVE-4411) negative test local_mapred_error_cache.q is failing trunk

2013-04-24 Thread Namit Jain (JIRA) 
Namit Jain created HIVE-4411:


 Summary: negative test local_mapred_error_cache.q is failing trunk
 Key: HIVE-4411
 URL: https://issues.apache.org/jira/browse/HIVE-4411
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4095) Add exchange partition in Hive

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4095:
-

Attachment: hive.4095.svn.thrift.patch

Attaching the svn patch after generating thrift files, since the parallel 
testing tool does not work with a git patch.

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, 
 hive.4095.svn.thrift.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4095) Add exchange partition in Hive

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4095:
-

Attachment: hive.4095.svn.thrift.patch.refresh

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, 
 hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4340) ORC should provide raw data size

2013-04-24 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640180#comment-13640180
 ] 

Namit Jain commented on HIVE-4340:
--

+1

 ORC should provide raw data size
 

 Key: HIVE-4340
 URL: https://issues.apache.org/jira/browse/HIVE-4340
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4340.1.patch.txt


 ORC's SerDe currently does nothing, and hence does not calculate a raw data 
 size.  WriterImpl, however, has enough information to provide one.
 WriterImpl should compute a raw data size for each row, aggregate them per 
 stripe and record it in the strip information, as RC currently does in its 
 key header, and allow the FileSinkOperator access to the size per row.
 FileSinkOperator should be able to get the raw data size from either the 
 SerDe or the RecordWriter when the RecordWriter can provide it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4324) ORC Turn off dictionary encoding when number of distinct keys is greater than threshold

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4324:
-

Status: Open  (was: Patch Available)

can you address Owen's comments ?

 ORC Turn off dictionary encoding when number of distinct keys is greater than 
 threshold
 ---

 Key: HIVE-4324
 URL: https://issues.apache.org/jira/browse/HIVE-4324
 Project: Hive
  Issue Type: Sub-task
  Components: Serializers/Deserializers
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4324.1.patch.txt


 Add a configurable threshold so that if the number of distinct values in a 
 string column is greater than that fraction of non-null values, dictionary 
 encoding is turned off.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4005) Column truncation

2013-04-24 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640186#comment-13640186
 ] 

Namit Jain commented on HIVE-4005:
--

Sorry, this one slipped under the radar.
Can you refresh one more time ?

 Column truncation
 -

 Key: HIVE-4005
 URL: https://issues.apache.org/jira/browse/HIVE-4005
 Project: Hive
  Issue Type: New Feature
  Components: CLI
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4005.1.patch.txt, HIVE-4005.2.patch.txt, 
 HIVE-4005.3.patch.txt, HIVE-4005.4.patch.txt, HIVE-4005.5.patch.txt


 Column truncation allows users to remove data for columns that are no longer 
 useful.
 This is done by removing the data for the column and setting the length of 
 the column data and related lengths to 0 in the RC file header.
 RC file was fixed to recognize columns with lengths of zero to be empty and 
 are treated as if the column doesn't exist in the data, a null is returned 
 for every value of that column in every row. This is the same thing that 
 happens when more columns are selected than exist in the file.
 A new command was added to the CLI
 TRUNCATE TABLE ... PARTITION ... COLUMNS ...
 This launches a map only job where each mapper rewrites a single file without 
 the unnecessary column data and the adjusted headers. It does not 
 uncompress/deserialize the data so it is much faster than rewriting the data 
 with NULLs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4005) Column truncation

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4005:
-

Status: Open  (was: Patch Available)

 Column truncation
 -

 Key: HIVE-4005
 URL: https://issues.apache.org/jira/browse/HIVE-4005
 Project: Hive
  Issue Type: New Feature
  Components: CLI
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4005.1.patch.txt, HIVE-4005.2.patch.txt, 
 HIVE-4005.3.patch.txt, HIVE-4005.4.patch.txt, HIVE-4005.5.patch.txt


 Column truncation allows users to remove data for columns that are no longer 
 useful.
 This is done by removing the data for the column and setting the length of 
 the column data and related lengths to 0 in the RC file header.
 RC file was fixed to recognize columns with lengths of zero to be empty and 
 are treated as if the column doesn't exist in the data, a null is returned 
 for every value of that column in every row. This is the same thing that 
 happens when more columns are selected than exist in the file.
 A new command was added to the CLI
 TRUNCATE TABLE ... PARTITION ... COLUMNS ...
 This launches a map only job where each mapper rewrites a single file without 
 the unnecessary column data and the adjusted headers. It does not 
 uncompress/deserialize the data so it is much faster than rewriting the data 
 with NULLs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4375) Single sourced multi insert consists of native and non-native table mixed throws NPE

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4375:
-

Status: Open  (was: Patch Available)

minor comments

 Single sourced multi insert consists of native and non-native table mixed 
 throws NPE
 

 Key: HIVE-4375
 URL: https://issues.apache.org/jira/browse/HIVE-4375
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-4375.D10329.1.patch


 CREATE TABLE src_x1(key string, value string);
 CREATE TABLE src_x2(key string, value string)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string);
 explain
 from src a
 insert overwrite table src_x1
 select key,value where a.key  0 AND a.key  50
 insert overwrite table src_x2
 select key,value where a.key  50 AND a.key  100;
 throws,
 {noformat}
 java.lang.NullPointerException
   at 
 org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.addStatsTask(GenMRFileSink1.java:236)
   at 
 org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:126)
   at 
 org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
   at 
 org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:87)
   at 
 org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:55)
   at 
 org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
   at 
 org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
   at 
 org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
   at 
 org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:101)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:8354)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8759)
   at 
 org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279)
   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433)
   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
   at 
 org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4340) ORC should provide raw data size

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4340:
-

Status: Open  (was: Patch Available)

The patch is not applying cleanly. Can you refresh ?

 ORC should provide raw data size
 

 Key: HIVE-4340
 URL: https://issues.apache.org/jira/browse/HIVE-4340
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
 Attachments: HIVE-4340.1.patch.txt


 ORC's SerDe currently does nothing, and hence does not calculate a raw data 
 size.  WriterImpl, however, has enough information to provide one.
 WriterImpl should compute a raw data size for each row, aggregate them per 
 stripe and record it in the strip information, as RC currently does in its 
 key header, and allow the FileSinkOperator access to the size per row.
 FileSinkOperator should be able to get the raw data size from either the 
 SerDe or the RecordWriter when the RecordWriter can provide it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4209:
-

Status: Open  (was: Patch Available)

comments

 Cache evaluation result of deterministic expression and reuse it
 

 Key: HIVE-4209
 URL: https://issues.apache.org/jira/browse/HIVE-4209
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-4209.D9585.1.patch, HIVE-4209.D9585.2.patch, 
 HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch


 For example, 
 {noformat}
 select key from src where key + 1  100 AND key + 1  200 limit 3;
 {noformat}
 key + 1 need not to be evaluated twice.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3781) not all meta events call metastore event listener

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3781:
-

Status: Open  (was: Patch Available)

The code changes look good, but the patch is not applying cleanly.
Can you refresh ?

 not all meta events call metastore event listener
 -

 Key: HIVE-3781
 URL: https://issues.apache.org/jira/browse/HIVE-3781
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.9.0
Reporter: Sudhanshu Arora
Assignee: Navis
 Attachments: hive.3781.3.patch, hive.3781.4.patch, 
 HIVE-3781.D7731.1.patch, HIVE-3781.D7731.2.patch, HIVE-3781.D7731.3.patch


 An event listener must be called for any DDL activity. For example, 
 create_index, drop_index today does not call metaevent listener. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4068:
-

Status: Open  (was: Patch Available)

comments

 Size of aggregation buffer which uses non-primitive type is not estimated 
 correctly
 ---

 Key: HIVE-4068
 URL: https://issues.apache.org/jira/browse/HIVE-4068
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, 
 HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch


 Currently, hive assumes an aggregation buffer which holds a map is occupying 
 just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can 
 be thrown (especially for 1k buffer). 
 workaround : set hive.map.aggr.hash.percentmemory=smaller value than 
 default(0.5)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3562:
-

Status: Open  (was: Patch Available)

can you refresh ?

 Some limit can be pushed down to map stage
 --

 Key: HIVE-3562
 URL: https://issues.apache.org/jira/browse/HIVE-3562
 Project: Hive
  Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
 Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch, 
 HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch


 Queries with limit clause (with reasonable number), for example
 {noformat}
 select * from src order by key limit 10;
 {noformat}
 makes operator tree, 
 TS-SEL-RS-EXT-LIMIT-FS
 But LIMIT can be partially calculated in RS, reducing size of shuffling.
 TS-SEL-RS(TOP-N)-EXT-LIMIT-FS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4181) Star argument without table alias for UDTF is not working

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4181:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Navis

 Star argument without table alias for UDTF is not working
 -

 Key: HIVE-4181
 URL: https://issues.apache.org/jira/browse/HIVE-4181
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
 Fix For: 0.12.0

 Attachments: HIVE-4181.D9453.1.patch


 select explode(*) as x from src; thorws
 {noformat}
 java.lang.NullPointerException
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genUDTFPlan(SemanticAnalyzer.java:5344)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2526)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2284)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:7079)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:7050)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:7792)
   at 
 org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8462)
   at 
 org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:259)
   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433)
   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
   at 
 org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3952) merge map-job followed by map-reduce job

2013-04-24 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3952:
-

Attachment: hive.3952.1.patch

 merge map-job followed by map-reduce job
 

 Key: HIVE-3952
 URL: https://issues.apache.org/jira/browse/HIVE-3952
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Namit Jain
Assignee: Vinod Kumar Vavilapalli
 Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, 
 HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, 
 HIVE-3952-20130424.txt


 Consider the query like:
 select count(*) FROM
 ( select idOne, idTwo, value FROM
   bigTable   
   JOIN
 
   smallTableOne on (bigTable.idOne = smallTableOne.idOne) 
   
   ) firstjoin 
 
 JOIN  
 
 smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
 where smallTableOne and smallTableTwo are smaller than 
 hive.auto.convert.join.noconditionaltask.size and
 hive.auto.convert.join.noconditionaltask is set to true.
 The joins are collapsed into mapjoins, and it leads to a map-only job
 (for the map-joins) followed by a map-reduce job (for the group by).
 Ideally, the map-only job should be merged with the following map-reduce job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job

2013-04-24 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641374#comment-13641374
 ] 

Namit Jain commented on HIVE-3952:
--

merge conflicts again - attached latest patch.

 merge map-job followed by map-reduce job
 

 Key: HIVE-3952
 URL: https://issues.apache.org/jira/browse/HIVE-3952
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Namit Jain
Assignee: Vinod Kumar Vavilapalli
 Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, 
 HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, 
 HIVE-3952-20130424.txt


 Consider the query like:
 select count(*) FROM
 ( select idOne, idTwo, value FROM
   bigTable   
   JOIN
 
   smallTableOne on (bigTable.idOne = smallTableOne.idOne) 
   
   ) firstjoin 
 
 JOIN  
 
 smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
 where smallTableOne and smallTableTwo are smaller than 
 hive.auto.convert.join.noconditionaltask.size and
 hive.auto.convert.join.noconditionaltask is set to true.
 The joins are collapsed into mapjoins, and it leads to a map-only job
 (for the map-joins) followed by a map-reduce job (for the group by).
 Ideally, the map-only job should be merged with the following map-reduce job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job

2013-04-24 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641375#comment-13641375
 ] 

Namit Jain commented on HIVE-3952:
--

https://reviews.facebook.net/D10533 just for the record

 merge map-job followed by map-reduce job
 

 Key: HIVE-3952
 URL: https://issues.apache.org/jira/browse/HIVE-3952
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Namit Jain
Assignee: Vinod Kumar Vavilapalli
 Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, 
 HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, 
 HIVE-3952-20130424.txt


 Consider the query like:
 select count(*) FROM
 ( select idOne, idTwo, value FROM
   bigTable   
   JOIN
 
   smallTableOne on (bigTable.idOne = smallTableOne.idOne) 
   
   ) firstjoin 
 
 JOIN  
 
 smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
 where smallTableOne and smallTableTwo are smaller than 
 hive.auto.convert.join.noconditionaltask.size and
 hive.auto.convert.join.noconditionaltask is set to true.
 The joins are collapsed into mapjoins, and it leads to a map-only job
 (for the map-joins) followed by a map-reduce job (for the group by).
 Ideally, the map-only job should be merged with the following map-reduce job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job

2013-04-24 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641383#comment-13641383
 ] 

Namit Jain commented on HIVE-3952:
--

+1

 merge map-job followed by map-reduce job
 

 Key: HIVE-3952
 URL: https://issues.apache.org/jira/browse/HIVE-3952
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Namit Jain
Assignee: Vinod Kumar Vavilapalli
 Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, 
 HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, 
 HIVE-3952-20130424.txt


 Consider the query like:
 select count(*) FROM
 ( select idOne, idTwo, value FROM
   bigTable   
   JOIN
 
   smallTableOne on (bigTable.idOne = smallTableOne.idOne) 
   
   ) firstjoin 
 
 JOIN  
 
 smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
 where smallTableOne and smallTableTwo are smaller than 
 hive.auto.convert.join.noconditionaltask.size and
 hive.auto.convert.join.noconditionaltask is set to true.
 The joins are collapsed into mapjoins, and it leads to a map-only job
 (for the map-joins) followed by a map-reduce job (for the group by).
 Ideally, the map-only job should be merged with the following map-reduce job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4018) MapJoin failing with Distributed Cache error

2013-04-23 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638926#comment-13638926
 ] 

Namit Jain commented on HIVE-4018:
--

will do

 MapJoin failing with Distributed Cache error
 

 Key: HIVE-4018
 URL: https://issues.apache.org/jira/browse/HIVE-4018
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.11.0
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.11.0

 Attachments: HIVE-4018.patch, hive.4018.test.2.patch, 
 HIVE-4018-test.patch


 When I'm a running a star join query after HIVE-3784, it is failing with 
 following error:
 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: 
 Load Distributed Cache Error
 2013-02-13 08:36:04,585 FATAL ExecMapper: 
 org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189)
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614)
   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
   at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:416)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
   at org.apache.hadoop.mapred.Child.main(Child.java:260)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4018) MapJoin failing with Distributed Cache error

2013-04-23 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638931#comment-13638931
 ] 

Namit Jain commented on HIVE-4018:
--

+1

 MapJoin failing with Distributed Cache error
 

 Key: HIVE-4018
 URL: https://issues.apache.org/jira/browse/HIVE-4018
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.11.0
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.11.0

 Attachments: HIVE-4018.patch, hive.4018.test.2.patch, 
 HIVE-4018-test.patch


 When I'm a running a star join query after HIVE-3784, it is failing with 
 following error:
 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: 
 Load Distributed Cache Error
 2013-02-13 08:36:04,585 FATAL ExecMapper: 
 org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189)
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614)
   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
   at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:416)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
   at org.apache.hadoop.mapred.Child.main(Child.java:260)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4018) MapJoin failing with Distributed Cache error

2013-04-23 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638934#comment-13638934
 ] 

Namit Jain commented on HIVE-4018:
--

[~amareshwari], can you load the latest patch (as an attachment) ?

 MapJoin failing with Distributed Cache error
 

 Key: HIVE-4018
 URL: https://issues.apache.org/jira/browse/HIVE-4018
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.11.0
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.11.0

 Attachments: HIVE-4018.patch, hive.4018.test.2.patch, 
 HIVE-4018-test.patch


 When I'm a running a star join query after HIVE-3784, it is failing with 
 following error:
 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: 
 Load Distributed Cache Error
 2013-02-13 08:36:04,585 FATAL ExecMapper: 
 org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189)
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614)
   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
   at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:416)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
   at org.apache.hadoop.mapred.Child.main(Child.java:260)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4018) MapJoin failing with Distributed Cache error

2013-04-23 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4018:
-

Status: Open  (was: Patch Available)

 MapJoin failing with Distributed Cache error
 

 Key: HIVE-4018
 URL: https://issues.apache.org/jira/browse/HIVE-4018
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.11.0
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.11.0

 Attachments: HIVE-4018-2.txt, HIVE-4018.patch, 
 hive.4018.test.2.patch, HIVE-4018-test.patch


 When I'm a running a star join query after HIVE-3784, it is failing with 
 following error:
 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: 
 Load Distributed Cache Error
 2013-02-13 08:36:04,585 FATAL ExecMapper: 
 org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189)
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614)
   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
   at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:416)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
   at org.apache.hadoop.mapred.Child.main(Child.java:260)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join

2013-04-23 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638973#comment-13638973
 ] 

Namit Jain commented on HIVE-3891:
--

comments

 physical optimizer changes for auto sort-merge join
 ---

 Key: HIVE-3891
 URL: https://issues.apache.org/jira/browse/HIVE-3891
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, 
 hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, 
 hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, 
 hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, 
 hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, 
 HIVE-3891_8.patch, hive.3891.9.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4018) MapJoin failing with Distributed Cache error

2013-04-23 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4018:
-

Fix Version/s: (was: 0.11.0)
   0.12.0
 Hadoop Flags: Reviewed
   Status: Patch Available  (was: Open)

Committed. Thanks Amareshwari

 MapJoin failing with Distributed Cache error
 

 Key: HIVE-4018
 URL: https://issues.apache.org/jira/browse/HIVE-4018
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.11.0
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.12.0

 Attachments: HIVE-4018-2.txt, HIVE-4018.patch, 
 hive.4018.test.2.patch, HIVE-4018-test.patch


 When I'm a running a star join query after HIVE-3784, it is failing with 
 following error:
 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: 
 Load Distributed Cache Error
 2013-02-13 08:36:04,585 FATAL ExecMapper: 
 org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189)
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614)
   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
   at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:416)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
   at org.apache.hadoop.mapred.Child.main(Child.java:260)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events

2013-04-23 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639284#comment-13639284
 ] 

Namit Jain commented on HIVE-4393:
--

+1

Running tests.

 Make the deleteData flag accessable from DropTable/Partition events
 ---

 Key: HIVE-4393
 URL: https://issues.apache.org/jira/browse/HIVE-4393
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Morgan Phillips
Assignee: Morgan Phillips
Priority: Minor
 Attachments: HIVE-4393.D10443.1.patch, HIVE-4393.D10443.2.patch


 On occasion, due to some error during a drop, information is removed from the 
 metastore but data, which should have been removed, remains intact on the 
 DFS.  In order to log such events via PreEvent and Event listeners a new 
 method 'getDeleteData' should be added to (Pre)DropPartitionEvent and 
 (Pre)DropTableEvent which returns the deleteData flag's value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events

2013-04-23 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4393:
-

Status: Patch Available  (was: Open)

 Make the deleteData flag accessable from DropTable/Partition events
 ---

 Key: HIVE-4393
 URL: https://issues.apache.org/jira/browse/HIVE-4393
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Morgan Phillips
Assignee: Morgan Phillips
Priority: Minor
 Attachments: HIVE-4393.D10443.1.patch, HIVE-4393.D10443.2.patch


 On occasion, due to some error during a drop, information is removed from the 
 metastore but data, which should have been removed, remains intact on the 
 DFS.  In order to log such events via PreEvent and Event listeners a new 
 method 'getDeleteData' should be added to (Pre)DropPartitionEvent and 
 (Pre)DropTableEvent which returns the deleteData flag's value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events

2013-04-23 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4393:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Morgan

 Make the deleteData flag accessable from DropTable/Partition events
 ---

 Key: HIVE-4393
 URL: https://issues.apache.org/jira/browse/HIVE-4393
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Morgan Phillips
Assignee: Morgan Phillips
Priority: Minor
 Fix For: 0.12.0

 Attachments: HIVE-4393.D10443.1.patch, HIVE-4393.D10443.2.patch


 On occasion, due to some error during a drop, information is removed from the 
 metastore but data, which should have been removed, remains intact on the 
 DFS.  In order to log such events via PreEvent and Event listeners a new 
 method 'getDeleteData' should be added to (Pre)DropPartitionEvent and 
 (Pre)DropTableEvent which returns the deleteData flag's value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4018) MapJoin failing with Distributed Cache error

2013-04-23 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4018:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 MapJoin failing with Distributed Cache error
 

 Key: HIVE-4018
 URL: https://issues.apache.org/jira/browse/HIVE-4018
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.11.0
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.12.0

 Attachments: HIVE-4018-2.txt, HIVE-4018.patch, 
 hive.4018.test.2.patch, HIVE-4018-test.patch


 When I'm a running a star join query after HIVE-3784, it is failing with 
 following error:
 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: 
 Load Distributed Cache Error
 2013-02-13 08:36:04,585 FATAL ExecMapper: 
 org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189)
   at 
 org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425)
   at 
 org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614)
   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
   at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:416)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
   at org.apache.hadoop.mapred.Child.main(Child.java:260)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-23 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640052#comment-13640052
 ] 

Namit Jain commented on HIVE-4300:
--

sure, running tests

 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
 # modified:   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote
 #
 # Untracked files:
 #   (use git add file... to include in what will be committed)
 #
 # serde/src/gen/thrift/gen-cpp/complex_constants.cpp
 # serde/src/gen/thrift/gen-cpp/complex_constants.h
 # serde/src/gen/thrift/gen-cpp/complex_types.cpp
 # serde/src/gen/thrift/gen-cpp/complex_types.h
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.h
 # serde/src/gen/thrift/gen-cpp/megastruct_types.cpp
 # serde/src/gen/thrift/gen-cpp

[jira] [Commented] (HIVE-4371) some issue with merging join trees

2013-04-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637782#comment-13637782
 ] 

Namit Jain commented on HIVE-4371:
--

Looks good.

Running tests

 some issue with merging join trees
 --

 Key: HIVE-4371
 URL: https://issues.apache.org/jira/browse/HIVE-4371
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Namit Jain
Assignee: Navis
 Attachments: HIVE-4371.D10323.1.patch, HIVE-4371.D10323.2.patch


 [~navis], I would really appreciate if you can take a look.
 I am attaching a testcase, for which in the optimizer the join context left
 aliases and right aliases do not look correct.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4396) Create a efficient method to create a deep copy of a mapredwork

2013-04-22 Thread Namit Jain (JIRA) 
Namit Jain created HIVE-4396:


 Summary: Create a efficient method to create a deep copy of a 
mapredwork 
 Key: HIVE-4396
 URL: https://issues.apache.org/jira/browse/HIVE-4396
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Namit Jain


In many optimization steps, we create a copy of the query plan and then change 
some parts of it.
Currently, the only reliable way to do so is to serialize and deserialize it, 
which is fairly expensive.
It would be very useful to have a cheaper method to do the same.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join

2013-04-22 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3891:
-

Attachment: hive.3891.15.patch

 physical optimizer changes for auto sort-merge join
 ---

 Key: HIVE-3891
 URL: https://issues.apache.org/jira/browse/HIVE-3891
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, 
 hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, 
 hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, 
 hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, 
 hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, 
 hive.3891.9.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events

2013-04-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637839#comment-13637839
 ] 

Namit Jain commented on HIVE-4393:
--

[~mrrrgn], when a patch is ready for review, please mark 'Submit Patch'.

 Make the deleteData flag accessable from DropTable/Partition events
 ---

 Key: HIVE-4393
 URL: https://issues.apache.org/jira/browse/HIVE-4393
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Morgan Phillips
Assignee: Morgan Phillips
Priority: Minor

 On occasion, due to some error during a drop, information is removed from the 
 metastore but data, which should have been removed, remains intact on the 
 DFS.  In order to log such events via PreEvent and Event listeners a new 
 method 'getDeleteData' should be added to (Pre)DropPartitionEvent and 
 (Pre)DropTableEvent which returns the deleteData flag's value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3952) merge map-job followed by map-reduce job

2013-04-22 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3952:
-

Status: Open  (was: Patch Available)

 merge map-job followed by map-reduce job
 

 Key: HIVE-3952
 URL: https://issues.apache.org/jira/browse/HIVE-3952
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Namit Jain
Assignee: Vinod Kumar Vavilapalli
 Attachments: HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, 
 HIVE-3952-20130301.txt, HIVE-3952-20130421.txt


 Consider the query like:
 select count(*) FROM
 ( select idOne, idTwo, value FROM
   bigTable   
   JOIN
 
   smallTableOne on (bigTable.idOne = smallTableOne.idOne) 
   
   ) firstjoin 
 
 JOIN  
 
 smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
 where smallTableOne and smallTableTwo are smaller than 
 hive.auto.convert.join.noconditionaltask.size and
 hive.auto.convert.join.noconditionaltask is set to true.
 The joins are collapsed into mapjoins, and it leads to a map-only job
 (for the map-joins) followed by a map-reduce job (for the group by).
 Ideally, the map-only job should be merged with the following map-reduce job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4371) some issue with merging join trees

2013-04-22 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4371:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed. Thanks Navis

 some issue with merging join trees
 --

 Key: HIVE-4371
 URL: https://issues.apache.org/jira/browse/HIVE-4371
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Namit Jain
Assignee: Navis
 Fix For: 0.12.0

 Attachments: HIVE-4371.D10323.1.patch, HIVE-4371.D10323.2.patch


 [~navis], I would really appreciate if you can take a look.
 I am attaching a testcase, for which in the optimizer the join context left
 aliases and right aliases do not look correct.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4106) SMB joins fail in multi-way joins

2013-04-22 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4106:
-

   Resolution: Fixed
Fix Version/s: 0.12.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Fixed as part of HIVE-4371

 SMB joins fail in multi-way joins
 -

 Key: HIVE-4106
 URL: https://issues.apache.org/jira/browse/HIVE-4106
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Namit Jain
Priority: Blocker
 Fix For: 0.12.0

 Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, 
 hive.4106.2.patch, HIVE-4106.patch


 I see array out of bounds exception in case of multi way smb joins. This is 
 related to changes that went in as part of HIVE-3403. This issue has been 
 discussed in HIVE-3891.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join

2013-04-22 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3891:
-

Attachment: hive.3891.16.patch

 physical optimizer changes for auto sort-merge join
 ---

 Key: HIVE-3891
 URL: https://issues.apache.org/jira/browse/HIVE-3891
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, 
 hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, 
 hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, 
 hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, 
 hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, 
 HIVE-3891_8.patch, hive.3891.9.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join

2013-04-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637908#comment-13637908
 ] 

Namit Jain commented on HIVE-3891:
--

Refreshed, addressed comments, ran tests.
Can you take a look again ?
cc [~ashutoshc]

 physical optimizer changes for auto sort-merge join
 ---

 Key: HIVE-3891
 URL: https://issues.apache.org/jira/browse/HIVE-3891
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, 
 hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, 
 hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, 
 hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, 
 hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, 
 HIVE-3891_8.patch, hive.3891.9.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4377) Add more comment to https://reviews.facebook.net/D1209 (HIVE-2340)

2013-04-22 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4377:
-

Status: Open  (was: Patch Available)

 Add more comment to https://reviews.facebook.net/D1209 (HIVE-2340)
 --

 Key: HIVE-4377
 URL: https://issues.apache.org/jira/browse/HIVE-4377
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Gang Tim Liu
Assignee: Navis
 Attachments: HIVE-4377.D10377.1.patch


 thanks a lot for addressing optimization in HIVE-2340. Awesome!
 Since we are developing at a very fast pace, it would be really useful to
 think about maintainability and testing of the large codebase. Highlights 
 which are applicable for D1209:
   1.  Javadoc for all public/private functions, except for
 setters/getters. For any complex function, clear examples (input/output)
 would really help.
   2.  Specially, for query optimizations, it might be a good idea to have
 a simple working query at the top, and the expected changes. For e.g..
 The operator tree for that query at each step, or a detailed explanation
 at the top.
   3.  If possible, the test name (.q file) where the function is being
 invoked, or the query which would potentially test that scenario, if it
 is a query processor change.
   4.  Comments in each test (.q file)­ that should include the jira
 number,  what is it trying to test. Assumptions about each query.
   5.  Reduce the output for each test ­ whenever query is outputting more
 than 10 results, it should have a reason. Otherwise, each query result
 should be bounded by 10 rows.
 thanks a lot

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events

2013-04-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638746#comment-13638746
 ] 

Namit Jain commented on HIVE-4393:
--

Can you address my earlier comments on https://reviews.facebook.net/D10425 ?

 Make the deleteData flag accessable from DropTable/Partition events
 ---

 Key: HIVE-4393
 URL: https://issues.apache.org/jira/browse/HIVE-4393
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Morgan Phillips
Assignee: Morgan Phillips
Priority: Minor
 Attachments: HIVE-4393.D10443.1.patch


 On occasion, due to some error during a drop, information is removed from the 
 metastore but data, which should have been removed, remains intact on the 
 DFS.  In order to log such events via PreEvent and Event listeners a new 
 method 'getDeleteData' should be added to (Pre)DropPartitionEvent and 
 (Pre)DropTableEvent which returns the deleteData flag's value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events

2013-04-22 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4393:
-

Status: Open  (was: Patch Available)

 Make the deleteData flag accessable from DropTable/Partition events
 ---

 Key: HIVE-4393
 URL: https://issues.apache.org/jira/browse/HIVE-4393
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Morgan Phillips
Assignee: Morgan Phillips
Priority: Minor
 Attachments: HIVE-4393.D10443.1.patch


 On occasion, due to some error during a drop, information is removed from the 
 metastore but data, which should have been removed, remains intact on the 
 DFS.  In order to log such events via PreEvent and Event listeners a new 
 method 'getDeleteData' should be added to (Pre)DropPartitionEvent and 
 (Pre)DropTableEvent which returns the deleteData flag's value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638747#comment-13638747
 ] 

Namit Jain commented on HIVE-4300:
--

[~roshan_naik], did you do anything different ?
or just 
ant thriftif -Dthrift.home=...
on a mac.

 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
 # modified:   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote
 #
 # Untracked files:
 #   (use git add file... to include in what will be committed)
 #
 # serde/src/gen/thrift/gen-cpp/complex_constants.cpp
 # serde/src/gen/thrift/gen-cpp/complex_constants.h
 # serde/src/gen/thrift/gen-cpp/complex_types.cpp
 # serde/src/gen/thrift/gen-cpp/complex_types.h
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.h
 # serde/src

[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638755#comment-13638755
 ] 

Namit Jain commented on HIVE-4300:
--

ok, I will compare the patch with my patch - it should be same, and then
merge it.

 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
 # modified:   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote
 #
 # Untracked files:
 #   (use git add file... to include in what will be committed)
 #
 # serde/src/gen/thrift/gen-cpp/complex_constants.cpp
 # serde/src/gen/thrift/gen-cpp/complex_constants.h
 # serde/src/gen/thrift/gen-cpp/complex_types.cpp
 # serde/src/gen/thrift/gen-cpp/complex_types.h
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.h
 # serde/src/gen/thrift/gen

[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638764#comment-13638764
 ] 

Namit Jain commented on HIVE-4300:
--

The results are same.

+1

 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
 # modified:   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote
 #
 # Untracked files:
 #   (use git add file... to include in what will be committed)
 #
 # serde/src/gen/thrift/gen-cpp/complex_constants.cpp
 # serde/src/gen/thrift/gen-cpp/complex_constants.h
 # serde/src/gen/thrift/gen-cpp/complex_types.cpp
 # serde/src/gen/thrift/gen-cpp/complex_types.h
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.h
 # serde/src/gen/thrift/gen-cpp/megastruct_types.cpp
 # serde/src/gen/thrift/gen

[jira] [Commented] (HIVE-4106) SMB joins fail in multi-way joins

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637524#comment-13637524
 ] 

Namit Jain commented on HIVE-4106:
--

[~ashutoshc], confirmed that the test still failed after HIVE-4371

 SMB joins fail in multi-way joins
 -

 Key: HIVE-4106
 URL: https://issues.apache.org/jira/browse/HIVE-4106
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Namit Jain
Priority: Blocker
 Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, 
 hive.4106.2.patch, HIVE-4106.patch


 I see array out of bounds exception in case of multi way smb joins. This is 
 related to changes that went in as part of HIVE-3403. This issue has been 
 discussed in HIVE-3891.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4342) NPE for query involving UNION ALL with nested JOIN and UNION ALL

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637537#comment-13637537
 ] 

Namit Jain commented on HIVE-4342:
--

+1

 NPE for query involving UNION ALL with nested JOIN and UNION ALL
 

 Key: HIVE-4342
 URL: https://issues.apache.org/jira/browse/HIVE-4342
 Project: Hive
  Issue Type: Bug
  Components: Logging, Metastore, Query Processor
Affects Versions: 0.9.0
 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0
Reporter: Mihir Kulkarni
Assignee: Navis
Priority: Critical
 Attachments: HIVE-4342.D10407.1.patch, HiveCommands.txt, Query.txt, 
 sourceData1.txt, sourceData2.txt


 UNION ALL query with JOIN in first part and another UNION ALL in second part 
 gives NPE.
 bq. JOIN
 UNION ALL
 bq. UNION ALL
 Attachments:
 1. HiveCommands.txt : command script to setup schema for query under 
 consideration.
 2. sourceData1.txt and sourceData2.txt : required for above command script.
 3. Query.txt : Exact query which produces NPE.
 NOTE: you will need to update path to sourceData1.txt and sourceData2.txt in 
 the HiveCommands.txt to suit your environment.
 Attached files contain the schema and exact query which fails on Hive 0.9.
 It is worthwhile to note that the same query executes successfully on Hive 
 0.7.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4342) NPE for query involving UNION ALL with nested JOIN and UNION ALL

2013-04-21 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4342:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed. Thanks Navis

 NPE for query involving UNION ALL with nested JOIN and UNION ALL
 

 Key: HIVE-4342
 URL: https://issues.apache.org/jira/browse/HIVE-4342
 Project: Hive
  Issue Type: Bug
  Components: Logging, Metastore, Query Processor
Affects Versions: 0.9.0
 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0
Reporter: Mihir Kulkarni
Assignee: Navis
Priority: Critical
 Attachments: HIVE-4342.D10407.1.patch, HiveCommands.txt, Query.txt, 
 sourceData1.txt, sourceData2.txt


 UNION ALL query with JOIN in first part and another UNION ALL in second part 
 gives NPE.
 bq. JOIN
 UNION ALL
 bq. UNION ALL
 Attachments:
 1. HiveCommands.txt : command script to setup schema for query under 
 consideration.
 2. sourceData1.txt and sourceData2.txt : required for above command script.
 3. Query.txt : Exact query which produces NPE.
 NOTE: you will need to update path to sourceData1.txt and sourceData2.txt in 
 the HiveCommands.txt to suit your environment.
 Attached files contain the schema and exact query which fails on Hive 0.9.
 It is worthwhile to note that the same query executes successfully on Hive 
 0.7.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4389) thrift files are re-generated by compiling

2013-04-21 Thread Namit Jain (JIRA) 
Namit Jain created HIVE-4389:


 Summary: thrift files are re-generated by compiling
 Key: HIVE-4389
 URL: https://issues.apache.org/jira/browse/HIVE-4389
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain


I am not sure what is going on, but there seems to be a bunch of thrift changes
if I perform ant thriftif.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4389) thrift files are re-generated by compiling

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637609#comment-13637609
 ] 

Namit Jain commented on HIVE-4389:
--

https://reviews.facebook.net/D10413

 thrift files are re-generated by compiling
 --

 Key: HIVE-4389
 URL: https://issues.apache.org/jira/browse/HIVE-4389
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: hive.4389.1.patch


 I am not sure what is going on, but there seems to be a bunch of thrift 
 changes
 if I perform ant thriftif.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4389) thrift files are re-generated by compiling

2013-04-21 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4389:
-

Attachment: hive.4389.1.patch

 thrift files are re-generated by compiling
 --

 Key: HIVE-4389
 URL: https://issues.apache.org/jira/browse/HIVE-4389
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: hive.4389.1.patch


 I am not sure what is going on, but there seems to be a bunch of thrift 
 changes
 if I perform ant thriftif.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637766#comment-13637766
 ] 

Namit Jain commented on HIVE-4300:
--

[~roshan_naik], I ran all the tests for HIVE-4389 (all of them ran fine other 
than leadlag.q,
which is also failing on trunk, for which I have filed a jira 
There, I had just performed:

ant thriftif -Dthrift.home=/usr/local
on my mac -- no local changes.


 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
 # modified:   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote
 #
 # Untracked files:
 #   (use git add file... to include in what will be committed)
 #
 # serde/src/gen/thrift/gen-cpp/complex_constants.cpp
 # serde/src/gen/thrift/gen-cpp/complex_constants.h
 # serde/src/gen/thrift/gen-cpp/complex_types.cpp
 # serde/src/gen/thrift/gen

[jira] [Created] (HIVE-4394) test leadlag.q fails

2013-04-21 Thread Namit Jain (JIRA) 
Namit Jain created HIVE-4394:


 Summary: test leadlag.q fails
 Key: HIVE-4394
 URL: https://issues.apache.org/jira/browse/HIVE-4394
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain


ant test -Dtestcase=TestCliDriver -Dqfile=leadlag.q fails.

cc [~rhbutani]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637766#comment-13637766
 ] 

Namit Jain edited comment on HIVE-4300 at 4/22/13 4:32 AM:
---

[~roshan_naik], I ran all the tests for HIVE-4389 (all of them ran fine other 
than leadlag.q,
which is also failing on trunk, for which I have filed a jira HIVE-4394.

There, I had just performed:

ant thriftif -Dthrift.home=/usr/local
on my mac -- no local changes.


  was (Author: namit):
[~roshan_naik], I ran all the tests for HIVE-4389 (all of them ran fine 
other than leadlag.q,
which is also failing on trunk, for which I have filed a jira 
There, I had just performed:

ant thriftif -Dthrift.home=/usr/local
on my mac -- no local changes.

  
 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote

[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637767#comment-13637767
 ] 

Namit Jain commented on HIVE-4300:
--

Did you do anything differently ?

 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php
 # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php
 # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
 # deleted:
 service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
 # modified:   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
 # modified:   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote
 #
 # Untracked files:
 #   (use git add file... to include in what will be committed)
 #
 # serde/src/gen/thrift/gen-cpp/complex_constants.cpp
 # serde/src/gen/thrift/gen-cpp/complex_constants.h
 # serde/src/gen/thrift/gen-cpp/complex_types.cpp
 # serde/src/gen/thrift/gen-cpp/complex_types.h
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp
 # serde/src/gen/thrift/gen-cpp/megastruct_constants.h
 # serde/src/gen/thrift/gen-cpp/megastruct_types.cpp
 # serde/src/gen

[jira] [Commented] (HIVE-4389) thrift files are re-generated by compiling

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637768#comment-13637768
 ] 

Namit Jain commented on HIVE-4389:
--

[~ashutoshc], the tests already ran fine for me.
I updated HIVE-4300.


 thrift files are re-generated by compiling
 --

 Key: HIVE-4389
 URL: https://issues.apache.org/jira/browse/HIVE-4389
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: hive.4389.1.patch


 I am not sure what is going on, but there seems to be a bunch of thrift 
 changes
 if I perform ant thriftif.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4389) thrift files are re-generated by compiling

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637769#comment-13637769
 ] 

Namit Jain commented on HIVE-4389:
--

For some reason, the patch on HIVE-4300 did not apply cleanly for me.
Should we just commit this instead ? It should be the same change

 thrift files are re-generated by compiling
 --

 Key: HIVE-4389
 URL: https://issues.apache.org/jira/browse/HIVE-4389
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: hive.4389.1.patch


 I am not sure what is going on, but there seems to be a bunch of thrift 
 changes
 if I perform ant thriftif.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date

2013-04-21 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637770#comment-13637770
 ] 

Namit Jain commented on HIVE-4300:
--

I got fewer changes than yours. For eg:

metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
In each of the above files, only change is inside comments. word 'optional' 
changed to 'required'


none of the above files changed.

 ant thriftif  generated code that is checkedin is not up-to-date
 

 Key: HIVE-4300
 URL: https://issues.apache.org/jira/browse/HIVE-4300
 Project: Hive
  Issue Type: Bug
  Components: Thrift API
Affects Versions: 0.10.0
Reporter: Roshan Naik
Assignee: Roshan Naik
 Attachments: HIVE-4300.2.patch, HIVE-4300.patch


 running 'ant thriftif -Dthrift.home=/usr/local'  on a freshly checkedout 
 trunk should be a no-op as per 
 [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode]
 However this is not the case. Some of files seem to be have been relocated or 
 the classes in them are now in a different file.
 Below is the git status showing the state after the command is run:
 # On branch trunk
 # Changes not staged for commit:
 #   (use git add/rm file... to update what will be committed)
 #   (use git checkout -- file... to discard changes in working directory)
 #
 # modified:   build.properties
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
 # modified:   
 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
 # deleted:
 metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
 # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php
 # modified:   
 metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 # modified:   
 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java
 # modified:   
 serde/src/gen/thrift/gen

[jira] [Commented] (HIVE-4106) SMB joins fail in multi-way joins

2013-04-20 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637201#comment-13637201
 ] 

Namit Jain commented on HIVE-4106:
--

I think it would still fail - I havent't tried it, but am nearly sure.
Basically, it assumes the left aliases and right aliases in a certain way, and 
that assumption has been changed by the join merge now.
It only matters in this peculiar case - when the table being joined 'd' does 
not show up in the join condition -- should not be a common case.

 SMB joins fail in multi-way joins
 -

 Key: HIVE-4106
 URL: https://issues.apache.org/jira/browse/HIVE-4106
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Namit Jain
Priority: Blocker
 Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, 
 hive.4106.2.patch, HIVE-4106.patch


 I see array out of bounds exception in case of multi way smb joins. This is 
 related to changes that went in as part of HIVE-3403. This issue has been 
 discussed in HIVE-3891.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4310) optimize count(distinct) with hive.map.groupby.sorted

2013-04-20 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4310:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed. Thanks Gang

 optimize count(distinct) with hive.map.groupby.sorted
 -

 Key: HIVE-4310
 URL: https://issues.apache.org/jira/browse/HIVE-4310
 Project: Hive
  Issue Type: Improvement
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: hive.4310.1.patch, hive.4310.1.patch-nohcat, 
 hive.4310.2.patch-nohcat, hive.4310.3.patch-nohcat, hive.4310.4.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4095) Add exchange partition in Hive

2013-04-19 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4095:
-

Status: Open  (was: Patch Available)

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3509) Exclusive locks are not acquired when using dynamic partitions

2013-04-19 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636297#comment-13636297
 ] 

Namit Jain commented on HIVE-3509:
--

comments

 Exclusive locks are not acquired when using dynamic partitions
 --

 Key: HIVE-3509
 URL: https://issues.apache.org/jira/browse/HIVE-3509
 Project: Hive
  Issue Type: Bug
  Components: Locking
Affects Versions: 0.9.0
Reporter: Matt Martin
Assignee: Matt Martin
 Attachments: HIVE-3509.1.patch.txt, HIVE-3509.D10065.1.patch, 
 HIVE-3509.D10065.2.patch, HIVE-3509.D10065.3.patch, HIVE-3509.D10065.4.patch


 If locking is enabled, the acquireReadWriteLocks() method in 
 org.apache.hadoop.hive.ql.Driver iterates through all of the input and output 
 entities of the query plan and attempts to acquire the appropriate locks.  In 
 general, it should acquire SHARED locks for all of the input entities and 
 exclusive locks for all of the output entities (see the Hive wiki page on 
 [locking|https://cwiki.apache.org/confluence/display/Hive/Locking] for more 
 detailed information).
 When the query involves dynamic partitions, the situation is a little more 
 subtle.  As the Hive wiki notes (see previous link):
 {quote}
 in some cases, the list of objects may not be known - for eg. in case of 
 dynamic partitions, the list of partitions being modified is not known at 
 compile time - so, the list is generated conservatively. Since the number of 
 partitions may not be known, an exclusive lock is taken on the table, or the 
 prefix that is known.
 {quote}
 After [HIVE-1781|https://issues.apache.org/jira/browse/HIVE-1781], the 
 observed behavior is no longer consistent with the behavior described above.  
 [HIVE-1781|https://issues.apache.org/jira/browse/HIVE-1781] appears to have 
 altered the logic so that SHARED locks are acquired instead of EXCLUSIVE 
 locks whenever the query involves dynamic partitions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4371) some issue with merging join trees

2013-04-18 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13634999#comment-13634999
 ] 

Namit Jain commented on HIVE-4371:
--

I am not sure about the last test case.
Why is left Alias (es) and right Alias (es) not correct for that ?

 some issue with merging join trees
 --

 Key: HIVE-4371
 URL: https://issues.apache.org/jira/browse/HIVE-4371
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Namit Jain
Assignee: Navis
 Attachments: HIVE-4371.D10323.1.patch


 [~navis], I would really appreciate if you can take a look.
 I am attaching a testcase, for which in the optimizer the join context left
 aliases and right aliases do not look correct.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join

2013-04-18 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3891:
-

Attachment: hive.3891.14.patch

 physical optimizer changes for auto sort-merge join
 ---

 Key: HIVE-3891
 URL: https://issues.apache.org/jira/browse/HIVE-3891
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, 
 hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, 
 hive.3891.13.patch, hive.3891.14.patch, hive.3891.1.patch, hive.3891.2.patch, 
 hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, 
 hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join

2013-04-18 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635005#comment-13635005
 ] 

Namit Jain commented on HIVE-3891:
--

[~ashutoshc], all the tests passed. Since this was accepted sometime back, can 
you take a look again ?

 physical optimizer changes for auto sort-merge join
 ---

 Key: HIVE-3891
 URL: https://issues.apache.org/jira/browse/HIVE-3891
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, 
 hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, 
 hive.3891.13.patch, hive.3891.14.patch, hive.3891.1.patch, hive.3891.2.patch, 
 hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, 
 hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4095) Add exchange partition in Hive

2013-04-18 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635108#comment-13635108
 ] 

Namit Jain commented on HIVE-4095:
--

more comments

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

hi

2013-04-18 Thread Namit Jain 
Hi,

Since we are developing at a very fast pace, it would be really useful to think 
about maintainability and testing of the large codebase.
Historically, we have not focussed on a few things, and they might soon bite 
us. I wanted to propose the following for all checkins:


  1.  Javadoc for all public/private functions, except for setters/getters. For 
any complex function, clear examples (input/output) would really help.
  2.  Convention for variable/function names – do we have any ?
  3.  If possible, the test name (.q file) where the function is being invoked, 
or the query which would potentially test that scenario, if it is a query 
processor change.
  4.  Specially, for query optimizations, it might be a good idea to have a 
simple working query at the top, and the expected changes. For e.g.. The 
operator tree for that query at each step, or a detailed explanation at the top.
  5.  Comments in each test (.q file)– that should include the jira number,  
what is it trying to test. Assumptions about each query.
  6.  Reduce the output for each test – whenever query is outputting more than 
10 results, it should have a reason. Otherwise, each query result should be 
bounded by 10 rows.

In general, focussing on a lot of comments in the code will go a long way for 
everyone to follow along.

Thanks,
-namit

Re: hi

2013-04-18 Thread Namit Jain 
Agreed.


On 4/18/13 9:19 PM, Jarek Jarcec Cecho jar...@apache.org wrote:

Hi Namit,
I like your proposal very much and I would take it a bit further:

   1.  ... For any complex function, clear examples (input/output) would
really help.

I'm concerned that examples in the code (comments) might very quickly
become obsolete as it can very easily happen that someone will change the
code without changing the example. What about using for this purpose
normal unit tests? Developers will still be able to see the expected
input/output, but in addition we will have automatic way how to detect
(possibly incompatible) changes. Please note that I'm not suggesting to
abandon the *.q file tests, just to also include unit tests for complex
methods.

Jarcec

On Thu, Apr 18, 2013 at 12:31:10PM +, Namit Jain wrote:
 Hi,
 
 Since we are developing at a very fast pace, it would be really useful
to think about maintainability and testing of the large codebase.
 Historically, we have not focussed on a few things, and they might soon
bite us. I wanted to propose the following for all checkins:
 
 
   1.  Javadoc for all public/private functions, except for
setters/getters. For any complex function, clear examples (input/output)
would really help.
   2.  Convention for variable/function names ­ do we have any ?
   3.  If possible, the test name (.q file) where the function is being
invoked, or the query which would potentially test that scenario, if it
is a query processor change.
   4.  Specially, for query optimizations, it might be a good idea to
have a simple working query at the top, and the expected changes. For
e.g.. The operator tree for that query at each step, or a detailed
explanation at the top.
   5.  Comments in each test (.q file)­ that should include the jira
number,  what is it trying to test. Assumptions about each query.
   6.  Reduce the output for each test ­ whenever query is outputting
more than 10 results, it should have a reason. Otherwise, each query
result should be bounded by 10 rows.
 
 In general, focussing on a lot of comments in the code will go a long
way for everyone to follow along.
 
 Thanks,
 -namit

Re: hi

2013-04-18 Thread Namit Jain 
Having said that, it might be difficult to write unit tests for operator
trees.
Might take more time initially - so, making it a constraint might slow us
down.


On 4/18/13 9:41 PM, Brock Noland br...@cloudera.com wrote:

Hi,

I like the proposal as well!

On Thu, Apr 18, 2013 at 10:49 AM, Jarek Jarcec Cecho
jar...@apache.orgwrote:

 Hi Namit,
 I like your proposal very much and I would take it a bit further:

1.  ... For any complex function, clear examples (input/output)
would
 really help.

 I'm concerned that examples in the code (comments) might very quickly
 become obsolete as it can very easily happen that someone will change
the
 code without changing the example. What about using for this purpose
normal
 unit tests? Developers will still be able to see the expected
input/output,
 but in addition we will have automatic way how to detect (possibly
 incompatible) changes. Please note that I'm not suggesting to abandon
the
 *.q file tests, just to also include unit tests for complex methods.


I'd be interested in including more unit tests as well. I like the
existing
q file test framework but when working on code I find unit tests which can
complete in less than a second or allows for faster iterations than
waiting
30 or so seconds for a q-file test to complete.

Brock

[jira] [Commented] (HIVE-4095) Add exchange partition in Hive

2013-04-18 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635424#comment-13635424
 ] 

Namit Jain commented on HIVE-4095:
--

+1

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, 
 HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, 
 HIVE-4095.part12.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4095) Add exchange partition in Hive

2013-04-18 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4095:
-

Attachment: hive.4095.1.patch

 Add exchange partition in Hive
 --

 Key: HIVE-4095
 URL: https://issues.apache.org/jira/browse/HIVE-4095
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Dheeraj Kumar Singh
 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, 
 HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, 
 HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join

2013-04-17 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3891:
-

Attachment: hive.3891.13.patch

 physical optimizer changes for auto sort-merge join
 ---

 Key: HIVE-3891
 URL: https://issues.apache.org/jira/browse/HIVE-3891
 Project: Hive
  Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
 Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, 
 hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, 
 hive.3891.13.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, 
 hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, 
 HIVE-3891_8.patch, hive.3891.9.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HIVE-4106) SMB joins fail in multi-way joins

2013-04-17 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633835#comment-13633835
 ] 

Namit Jain edited comment on HIVE-4106 at 4/17/13 6:54 AM:
---

[~ashutoshc], can you take a look ?

  was (Author: namit):
[~ashutoshc], can you taje a look ?
  
 SMB joins fail in multi-way joins
 -

 Key: HIVE-4106
 URL: https://issues.apache.org/jira/browse/HIVE-4106
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Namit Jain
Priority: Blocker
 Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, 
 hive.4106.2.patch, HIVE-4106.patch


 I see array out of bounds exception in case of multi way smb joins. This is 
 related to changes that went in as part of HIVE-3403. This issue has been 
 discussed in HIVE-3891.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4106) SMB joins fail in multi-way joins

2013-04-17 Thread Namit Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-4106:
-

Assignee: Namit Jain  (was: Vikram Dixit K)
  Status: Patch Available  (was: Open)

 SMB joins fail in multi-way joins
 -

 Key: HIVE-4106
 URL: https://issues.apache.org/jira/browse/HIVE-4106
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Namit Jain
Priority: Blocker
 Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, 
 hive.4106.2.patch, HIVE-4106.patch


 I see array out of bounds exception in case of multi way smb joins. This is 
 related to changes that went in as part of HIVE-3403. This issue has been 
 discussed in HIVE-3891.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4106) SMB joins fail in multi-way joins

2013-04-17 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633835#comment-13633835
 ] 

Namit Jain commented on HIVE-4106:
--

[~ashutoshc], can you taje a look ?

 SMB joins fail in multi-way joins
 -

 Key: HIVE-4106
 URL: https://issues.apache.org/jira/browse/HIVE-4106
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Namit Jain
Priority: Blocker
 Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, 
 hive.4106.2.patch, HIVE-4106.patch


 I see array out of bounds exception in case of multi way smb joins. This is 
 related to changes that went in as part of HIVE-3403. This issue has been 
 discussed in HIVE-3891.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

  1   2   3   4   5   6   7   8   9   10   >