[
https://issues.apache.org/jira/browse/HIVE-21167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773816#comment-16773816
]
Hive QA commented on HIVE-21167:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12959524/HIVE-21167.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15809 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16175/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16175/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16175/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12959524 - PreCommit-HIVE-Build
> Bucketing: Bucketing version 1 is incorrectly partitioning data
> ---------------------------------------------------------------
>
> Key: HIVE-21167
> URL: https://issues.apache.org/jira/browse/HIVE-21167
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.1.1
> Reporter: Vaibhav Gumashta
> Assignee: Deepak Jaiswal
> Priority: Major
> Attachments: HIVE-21167.1.patch, HIVE-21167.2.patch
>
>
> Using murmur hash for bucketing columns was introduced in HIVE-18910,
> following which {{'bucketing_version'='1'}} stands for the old behaviour
> (where for example integer columns were partitioned based on mod values).
> Looks like we have a bug in the old bucketing scheme now. I could repro it
> when modified the existing schema using an alter table add column and adding
> new data. Repro:
> {code}
> 0: jdbc:hive2://localhost:10010> create transactional table acid_ptn_bucket1
> (a int, b int) partitioned by(ds string) clustered by (a) into 2 buckets
> stored as ORC TBLPROPERTIES('bucketing_version'='1', 'transactional'='true',
> 'transactional_properties'='default');
> No rows affected (0.418 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
> values(1,2,'today'),(1,3,'today'),(1,4,'yesterday'),(2,2,'yesterday'),(2,3,'today'),(2,4,'today');
> 6 rows affected (3.695 seconds)
> {code}
> Data from ORC file (data as expected):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00000
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 0,
> "currentTransaction": 1, "row": {"a": 2, "b": 4}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 1,
> "currentTransaction": 1, "row": {"a": 2, "b": 3}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00001
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 0,
> "currentTransaction": 1, "row": {"a": 1, "b": 3}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 1,
> "currentTransaction": 1, "row": {"a": 1, "b": 2}}
> {code}
> Modifying table schema and inserting new data:
> {code}
> 0: jdbc:hive2://localhost:10010> alter table acid_ptn_bucket1 add columns(c
> int);
> No rows affected (0.541 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
> values(3,2,1000,'yesterday'),(3,3,1001,'today'),(3,4,1002,'yesterday'),(4,2,1003,'today'),
> (4,3,1004,'yesterday'),(4,4,1005,'today');
> 6 rows affected (3.699 seconds)
> {code}
> Data from ORC file (wrong partitioning):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00000
> {"operation": 0, "originalTransaction": 3, "bucket": 536870912, "rowId": 0,
> "currentTransaction": 3, "row": {"a": 3, "b": 3, "c": 1001}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00001
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 0,
> "currentTransaction": 3, "row": {"a": 4, "b": 4, "c": 1005}}
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 1,
> "currentTransaction": 3, "row": {"a": 4, "b": 2, "c": 1003}}
> {code}
> As seen above, the expected behaviour is that new data with column 'a' being
> 3 should go to bucket1 and column 'a' being 4 should go to bucket0, but the
> partitioning is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
