date:20110725

[jira] [Commented] (HIVE-2283) Backtracking real column names for EXPLAIN output

2011-07-25 Thread Navis (JIRA)

[
https://issues.apache.org/jira/browse/HIVE-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070328#comment-13070328
]

Navis commented on HIVE-2283:
-

This patch would change the output of the explain query and, doubtedly, would
make fail other test cases. Making some option switching the explain mode would
be appropriate, but doing that was not so clean (redundant code, etc.) as it
was seemed to be.

I've just laid this down and waiting opinion of others.

Backtracking real column names for EXPLAIN output
-

Key: HIVE-2283
URL: https://issues.apache.org/jira/browse/HIVE-2283
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.8.0
Reporter: Navis
Priority: Minor
Attachments: HIVE-2283.1.patch, HIVE-2283.2.patch,
HIVE-2283.test.patch

GUI people suggested that showing real column names for result of EXPLAIN
statement would make customers feel more comfortable with HIVE. I agreed and
working on it.
{code}
a. current EXPLAIN
Select Operator
expressions:
expr: _col10
type: int
expr: _col17
type: string
Group By Operator
keys:
expr: _col0
type: int
expr: _col17
type: int
b. suggested EXPLAIN
Select Operator
expressions: _col10=t2.key_int1, _col17=upper(t1.key_int1),
_col22=t3.key_string2
Group By Operator
keys: _col10=t2.key_int1, _col17=upper(t1.key_int1)
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: HIVE-2246: Dedupe tables' column schemas from partitions in the metastore db

2011-07-25 Thread Ning Zhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1183/#review1176
---



trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql
https://reviews.apache.org/r/1183/#comment2467

is the CHARSET (latin1) the same as SDS? This will require the user's 
comments to be in latin1 which prevents UTF chars.



trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql
https://reviews.apache.org/r/1183/#comment2466

can you also add migration script for derby? we support derby as a default 
metastore RDBMS as well. 



trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
https://reviews.apache.org/r/1183/#comment2468

here do you check if the 'alter table' command changes the schema (columns 
definition)? If it just set a table property, then you don't need to create a 
new ColumnDescriptor right?

Also if a table's schema got changed, a new CD will be created, but the old 
partition will still have the old CDs. When we query the old partition, do we 
use the old partitons's CD or the table's CD? 

Also in the above case, when you run 'desc table partition 
old_partition', do you return the old partition's CD or the table's CD? 


- Ning


On 2011-07-22 05:30:29, Sohan Jain wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/1183/
 ---
 
 (Updated 2011-07-22 05:30:29)
 
 
 Review request for hive, Ning Zhang and Paul Yang.
 
 
 Summary
 ---
 
 This patch tries to make minimal changes to the API while keeping migration 
 short and somewhat easy to revert.
 
 The new schema can be described as follows:
 - CDS is a table corresponding to Column Descriptor objects.  Currently, it 
 only stores a CD_ID.
 - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns.  A 
 Column Descriptor holds a list of columns.  COLUMNS_V2 has a foreign key to 
 the CD_ID to which it belongs.
 - SDS was modified to reference a Column Descriptor. So SDS now has a foreign 
 key to a CD_ID which describes its columns.
 
 During migration, we create Column Descriptors for tables in a 
 straightforward manner: their columns are now just wrapped inside a column 
 descriptor.  The SDS of partitions use their parent table's column 
 descriptor, since currently a partition and its table share the same list of 
 columns.
 
 When altering or adding a partition, give it it's parent table's column 
 descriptor IF the columns they describe are the same.  Otherwise, create a 
 new column descriptor for its columns.
 
 When adding or altering a table, create a new column descriptor every time.
 
 Whenever you drop a storage descriptor (e.g, when dropping tables or 
 partitions), check to see if the related column descriptor has any other 
 references in the table.  That is, check to see if any other storage 
 descriptors point to that column descriptor.  If none do, then delete that 
 column descriptor.  This check is in place so we don't have unreferenced 
 column descriptors and columns hanging around after schema evolution for 
 tables.
 
 
 This addresses bug HIVE-2246.
 https://issues.apache.org/jira/browse/HIVE-2246
 
 
 Diffs
 -
 
   trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql PRE-CREATION 
   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
 1148945 
   
 trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnDescriptor.java
  PRE-CREATION 
   
 trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java
  1148945 
   trunk/metastore/src/model/package.jdo 1148945 
 
 Diff: https://reviews.apache.org/r/1183/diff
 
 
 Testing
 ---
 
 Passes facebook's regression testing and all existing test cases.  In one 
 instance, before migration, the overhead involved with storage descriptors 
 and columns was ~11 GB.  After migration, the overhead was ~1.5 GB.
 
 
 Thanks,
 
 Sohan

[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db

2011-07-25 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070337#comment-13070337
]

jirapos...@reviews.apache.org commented on HIVE-2246:
-

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1183/#review1176
---

trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql
https://reviews.apache.org/r/1183/#comment2467

is the CHARSET (latin1) the same as SDS? This will require the user's
comments to be in latin1 which prevents UTF chars.

trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql
https://reviews.apache.org/r/1183/#comment2466

can you also add migration script for derby? we support derby as a default
metastore RDBMS as well.

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
https://reviews.apache.org/r/1183/#comment2468

here do you check if the 'alter table' command changes the schema (columns
definition)? If it just set a table property, then you don't need to create a
new ColumnDescriptor right?

Also if a table's schema got changed, a new CD will be created, but the old
partition will still have the old CDs. When we query the old partition, do we
use the old partitons's CD or the table's CD?

Also in the above case, when you run 'desc table partition
old_partition', do you return the old partition's CD or the table's CD?

- Ning

On 2011-07-22 05:30:29, Sohan Jain wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1183/
bq. ---
bq.
bq. (Updated 2011-07-22 05:30:29)
bq.
bq.
bq. Review request for hive, Ning Zhang and Paul Yang.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This patch tries to make minimal changes to the API while keeping
migration short and somewhat easy to revert.
bq.
bq. The new schema can be described as follows:
bq. - CDS is a table corresponding to Column Descriptor objects. Currently,
it only stores a CD_ID.
bq. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns.
A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to
the CD_ID to which it belongs.
bq. - SDS was modified to reference a Column Descriptor. So SDS now has a
foreign key to a CD_ID which describes its columns.
bq.
bq. During migration, we create Column Descriptors for tables in a
straightforward manner: their columns are now just wrapped inside a column
descriptor. The SDS of partitions use their parent table's column descriptor,
since currently a partition and its table share the same list of columns.
bq.
bq. When altering or adding a partition, give it it's parent table's column
descriptor IF the columns they describe are the same. Otherwise, create a new
column descriptor for its columns.
bq.
bq. When adding or altering a table, create a new column descriptor every time.
bq.
bq. Whenever you drop a storage descriptor (e.g, when dropping tables or
partitions), check to see if the related column descriptor has any other
references in the table. That is, check to see if any other storage
descriptors point to that column descriptor. If none do, then delete that
column descriptor. This check is in place so we don't have unreferenced column
descriptors and columns hanging around after schema evolution for tables.
bq.
bq.
bq. This addresses bug HIVE-2246.
bq. https://issues.apache.org/jira/browse/HIVE-2246
bq.
bq.
bq. Diffs
bq. -
bq.
bq.trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql
PRE-CREATION
bq.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
1148945
bq.
trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnDescriptor.java
PRE-CREATION
bq.
trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java
1148945
bq.trunk/metastore/src/model/package.jdo 1148945
bq.
bq. Diff: https://reviews.apache.org/r/1183/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Passes facebook's regression testing and all existing test cases. In one
instance, before migration, the overhead involved with storage descriptors and
columns was ~11 GB. After migration, the overhead was ~1.5 GB.
bq.
bq.
bq. Thanks,
bq.
bq. Sohan
bq.
bq.

60 matches

Mail list logo