[jira] [Commented] (HIVE-14633) #.of Files in a partition ! = #.Of buckets in a partitioned,bucketed table
[ https://issues.apache.org/jira/browse/HIVE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449148#comment-15449148 ] Abhishek Somani commented on HIVE-14633: I think number of mappers can be controlled via other means like split size configurations, using CombineHiveInputFormat etc. Is this a Tez usecase? Tez does split grouping as well which should lead to lesser mappers. > #.of Files in a partition ! = #.Of buckets in a partitioned,bucketed table > -- > > Key: HIVE-14633 > URL: https://issues.apache.org/jira/browse/HIVE-14633 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 > Environment: HDP 2.3.2 >Reporter: Hanu > > Ideally the number of files should be equal to number of buckets declared in > a table DDL. It is working fine whenever an initial insert or every insert > overwrite is performed. But, insert into hive bucketed table is creating > extra files. > ex: > # of Buckets = 4 > No. of files after Initial insert --> 4 > No. of files after 2nd insert --> 8 > No. of files after 3rd insert --> 12 > No. of files after n insert --> n* # of Buckets. > First insert list : > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0 > -rwxrwxrwx 3 hvallur hdfs308 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0 > 2nd Insert: > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs308 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0 > -rwxrwxrwx 3 hvallur hdfs302 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0_copy_1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14633) #.of Files in a partition ! = #.Of buckets in a partitioned,bucketed table
[ https://issues.apache.org/jira/browse/HIVE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448135#comment-15448135 ] Abhishek Somani commented on HIVE-14633: I think this is expected. Insert into will just create those copy files you see, with the same bucket id as seen above. This is not expected to affect any functionality and hive takes care of those copies correctly. Others can confirm. Do you seen any functionality broken due to this? > #.of Files in a partition ! = #.Of buckets in a partitioned,bucketed table > -- > > Key: HIVE-14633 > URL: https://issues.apache.org/jira/browse/HIVE-14633 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 > Environment: HDP 2.3.2 >Reporter: Hanu > > Ideally the number of files should be equal to number of buckets declared in > a table DDL. It is working fine whenever an initial insert or every insert > overwrite is performed. But, insert into hive bucketed table is creating > extra files. > ex: > # of Buckets = 4 > No. of files after Initial insert --> 4 > No. of files after 2nd insert --> 8 > No. of files after 3rd insert --> 12 > No. of files after n insert --> n* # of Buckets. > First insert list : > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0 > -rwxrwxrwx 3 hvallur hdfs308 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0 > 2nd Insert: > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs308 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0 > -rwxrwxrwx 3 hvallur hdfs302 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0_copy_1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14633) #.of Files in a partition ! = #.Of buckets in a partitioned,bucketed table
[ https://issues.apache.org/jira/browse/HIVE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448135#comment-15448135 ] Abhishek Somani edited comment on HIVE-14633 at 8/30/16 5:54 AM: - Isn't this expected? Insert into will just create those copy files you see, with the same bucket id as seen above. This is not expected to affect any functionality and hive takes care of those copies correctly. Others can confirm. Do you seen any functionality broken due to this? was (Author: asomani): I think this is expected. Insert into will just create those copy files you see, with the same bucket id as seen above. This is not expected to affect any functionality and hive takes care of those copies correctly. Others can confirm. Do you seen any functionality broken due to this? > #.of Files in a partition ! = #.Of buckets in a partitioned,bucketed table > -- > > Key: HIVE-14633 > URL: https://issues.apache.org/jira/browse/HIVE-14633 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 > Environment: HDP 2.3.2 >Reporter: Hanu > > Ideally the number of files should be equal to number of buckets declared in > a table DDL. It is working fine whenever an initial insert or every insert > overwrite is performed. But, insert into hive bucketed table is creating > extra files. > ex: > # of Buckets = 4 > No. of files after Initial insert --> 4 > No. of files after 2nd insert --> 8 > No. of files after 3rd insert --> 12 > No. of files after n insert --> n* # of Buckets. > First insert list : > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0 > -rwxrwxrwx 3 hvallur hdfs308 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0 > 2nd Insert: > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/00_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/01_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs308 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0 > -rwxrwxrwx 3 hvallur hdfs302 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/02_0_copy_1 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:42 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0 > -rwxrwxrwx 3 hvallur hdfs 49 2016-08-25 12:47 > hdfs://dshdp-dev-cluster/apps/hive/warehouse/upsert_testing.db/test3/lname=vr/03_0_copy_1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated HIVE-15390: --- Attachment: HIVE-15390.patch Removing the resetting of the split's maxOffset to Long.MAX_VALUE > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732553#comment-15732553 ] Abhishek Somani edited comment on HIVE-15390 at 12/8/16 3:44 PM: - Attaching a patch removing the resetting of the split's maxOffset to Long.MAX_VALUE was (Author: asomani): Removing the resetting of the split's maxOffset to Long.MAX_VALUE > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[
https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732538#comment-15732538
]
Abhishek Somani commented on HIVE-15390:
I think this is happening due to a bug in OrcRawRecordMerger initialization
where we reset the split's length to Long.MAX_VALUE.
{code:java}
if (isOriginal) {
options = options.clone();
options.range(options.getOffset(), Long.MAX_VALUE);
pair = new OriginalReaderPair(key, reader, bucket, minKey, maxKey,
options);
}
{code}
This causes the reader to incorrectly assume that all the stripes from the
start offset to the end of the file is it's responsibility to read. In
RecordReaderImpl.java:
{code:java}
long offset = options.getOffset();
long maxOffset = options.getMaxOffset();
for(StripeInformation stripe: fileReader.getStripes()) {
long stripeStart = stripe.getOffset();
if (offset > stripeStart) {
skippedRows += stripe.getNumberOfRows();
} else if (stripeStart < maxOffset) {
this.stripes.add(stripe);
rows += stripe.getNumberOfRows();
}
}
{code}
Although the task does not make the mistake of reading data that it should not
be reading because it also relies on the min and max row key to make that
decision, in some cases it does up reading the footers of all the stripes,
increasing task completion time.
Once all the rows from a stripe have been read,
RecordReaderImpl.advanceToNextRow() is invoked which reads the footer of the
next stripe and evaluates the sarg against the rowgroups in that stripe using
the metadata in the footer. If sarg evaluation fails for all the rowgroups in
the stripe, it moves on the next stripe and repeats the same process there,
ending up reading stripe footers for all the stripes in the file.
Is my understanding correct?
> Orc reader unnecessarily reading stripe footers with
> hive.optimize.index.filter set to true
> ---
>
> Key: HIVE-15390
> URL: https://issues.apache.org/jira/browse/HIVE-15390
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 1.2.1
>Reporter: Abhishek Somani
>Assignee: Abhishek Somani
>
> In a split given to a task, the task's orc reader is unnecessarily reading
> stripe footers for stripes that are not its responsibility to read. This is
> happening with hive.optimize.index.filter set to true.
> Assuming one split per task(no tez grouping considered), a task should not
> need to read beyond the split's end offset. Even in some split computation
> strategies where a split's end offset can be in the middle of a stripe, it
> should not need to read more than one stripe beyond the split's end offset(to
> fully read a stripe that started in it). However I see that some tasks make
> unnecessary filesystem calls to read all the stripe footers in a file from
> the split start offset till the end of the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732553#comment-15732553 ] Abhishek Somani edited comment on HIVE-15390 at 12/8/16 3:46 PM: - Attaching a patch removing the resetting of the maxOffset to Long.MAX_VALUE was (Author: asomani): Attaching a patch removing the resetting of the split's maxOffset to Long.MAX_VALUE > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated HIVE-15390: --- Attachment: (was: HIVE-15390.patch) > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated HIVE-15390: --- Attachment: HIVE-15390.patch > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820473#comment-15820473 ] Abhishek Somani commented on HIVE-15390: [~prasanth_j] [~rajesh.balamohan] can you please review. > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.1.patch, HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820473#comment-15820473 ] Abhishek Somani edited comment on HIVE-15390 at 1/12/17 8:30 AM: - [~prasanth_j] [~rajesh.balamohan] [~gopalv] can you please review. was (Author: asomani): [~prasanth_j] [~rajesh.balamohan] can you please review. > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.1.patch, HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated HIVE-15390: --- Status: Open (was: Patch Available) Trying to trigger precommit tests. > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.1.patch, HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated HIVE-15390: --- Attachment: HIVE-15390.1.patch > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.1.patch, HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated HIVE-15390: --- Status: Patch Available (was: Open) > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.1.patch, HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated HIVE-15390: --- Status: Patch Available (was: Open) > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779695#comment-15779695 ] Abhishek Somani commented on HIVE-15390: Review Board: https://reviews.apache.org/r/55045/ Not sure why the precommit tests are not running. Uploading the same patch with a patch number to try and trigger the tests. > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15390) Orc reader unnecessarily reading stripe footers with hive.optimize.index.filter set to true
[ https://issues.apache.org/jira/browse/HIVE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779695#comment-15779695 ] Abhishek Somani edited comment on HIVE-15390 at 12/27/16 6:03 AM: -- Review Board: https://reviews.apache.org/r/55046 Not sure why the precommit tests are not running. Uploading the same patch with a patch number to try and trigger the tests. was (Author: asomani): Review Board: https://reviews.apache.org/r/55045/ Not sure why the precommit tests are not running. Uploading the same patch with a patch number to try and trigger the tests. > Orc reader unnecessarily reading stripe footers with > hive.optimize.index.filter set to true > --- > > Key: HIVE-15390 > URL: https://issues.apache.org/jira/browse/HIVE-15390 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Abhishek Somani >Assignee: Abhishek Somani > Attachments: HIVE-15390.1.patch, HIVE-15390.patch > > > In a split given to a task, the task's orc reader is unnecessarily reading > stripe footers for stripes that are not its responsibility to read. This is > happening with hive.optimize.index.filter set to true. > Assuming one split per task(no tez grouping considered), a task should not > need to read beyond the split's end offset. Even in some split computation > strategies where a split's end offset can be in the middle of a stripe, it > should not need to read more than one stripe beyond the split's end offset(to > fully read a stripe that started in it). However I see that some tasks make > unnecessary filesystem calls to read all the stripe footers in a file from > the split start offset till the end of the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14487) Add REBUILD statement for materialized views
[
https://issues.apache.org/jira/browse/HIVE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078518#comment-16078518
]
Abhishek Somani commented on HIVE-14487:
[~jcamachorodriguez] Is this being worked on? From your comment on 26/Aug/16 I
see this was almost ready to go. Has it gone in already?
> Add REBUILD statement for materialized views
>
>
> Key: HIVE-14487
> URL: https://issues.apache.org/jira/browse/HIVE-14487
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>
> Support for rebuilding existing materialized views. The statement is the
> following:
> {code:sql}
> ALTER MATERIALIZED VIEW [db_name.]materialized_view_name REBUILD;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16907) "INSERT INTO" overwrite old data when destination table encapsulated by backquote
[
https://issues.apache.org/jira/browse/HIVE-16907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260596#comment-16260596
]
Abhishek Somani commented on HIVE-16907:
I ran into this issue as well. Do we plan to fix it?
> "INSERT INTO" overwrite old data when destination table encapsulated by
> backquote
>
>
> Key: HIVE-16907
> URL: https://issues.apache.org/jira/browse/HIVE-16907
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.1.0, 2.1.1
>Reporter: Nemon Lou
>Assignee: Bing Li
> Attachments: HIVE-16907.1.patch
>
>
> A way to reproduce:
> {noformat}
> create database tdb;
> use tdb;
> create table t1(id int);
> create table t2(id int);
> explain insert into `tdb.t1` select * from t2;
> {noformat}
> {noformat}
> +---+
> |
> Explain |
> +---+
> | STAGE DEPENDENCIES:
> |
> | Stage-1 is a root stage
> |
> | Stage-6 depends on stages: Stage-1 , consists of Stage-3, Stage-2,
> Stage-4 |
> | Stage-3
> |
> | Stage-0 depends on stages: Stage-3, Stage-2, Stage-5
> |
> | Stage-2
> |
> | Stage-4
> |
> | Stage-5 depends on stages: Stage-4
> |
> |
> |
> | STAGE PLANS:
> |
> | Stage: Stage-1
> |
> | Map Reduce
> |
> | Map Operator Tree:
> |
> | TableScan
> |
> | alias: t2
> |
> | Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
> stats: NONE |
> | Select Operator
> |
> | expressions: id (type: int)
> |
> | outputColumnNames: _col0
> |
> | Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
> stats: NONE |
> | File Output Operator
> |
> | compressed: false
> |
> | Statistics: Num rows: 0 Data size: 0 Basic stats: NONE
> Column stats: NONE
[jira] [Commented] (HIVE-20220) Incorrect result when hive.groupby.skewindata is enabled
[ https://issues.apache.org/jira/browse/HIVE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591155#comment-16591155 ] Abhishek Somani commented on HIVE-20220: To add to Ganesha's point above, this is an issue with Tez as well. So this is a very pertinent issue. > Incorrect result when hive.groupby.skewindata is enabled > > > Key: HIVE-20220 > URL: https://issues.apache.org/jira/browse/HIVE-20220 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 3.0.0 >Reporter: Ganesha Shreedhara >Assignee: Ganesha Shreedhara >Priority: Major > Attachments: HIVE-20220.2.patch, HIVE-20220.patch > > > hive.groupby.skewindata makes use of rand UDF to randomly distribute grouped > by keys to the reducers and hence avoids overloading a single reducer when > there is a skew in data. > This random distribution of keys is buggy when the reducer fails to fetch the > mapper output due to a faulty datanode or any other reason. When reducer > finds that it can't fetch mapper output, it sends a signal to Application > Master to reattempt the corresponding map task. The reattempted map task will > now get the different random value from rand function and hence the keys that > gets distributed now to the reducer will not be same as the previous run. > > *Steps to reproduce:* > create table test(id int); > insert into test values > (1),(2),(2),(3),(3),(3),(4),(4),(4),(4),(5),(5),(5),(5),(5),(6),(6),(6),(6),(6),(6),(7),(7),(7),(7),(7),(7),(7),(7),(8),(8),(8),(8),(8),(8),(8),(8),(9),(9),(9),(9),(9),(9),(9),(9),(9); > SET hive.groupby.skewindata=true; > SET mapreduce.reduce.reduces=2; > //Add a debug port for reducer > select count(1) from test group by id; > //Remove mapper's intermediate output file when map stage is completed and > one out of 2 reduce tasks is completed and then continue the run. This causes > 2nd reducer to send event to Application Master to rerun the map task. > The following is the expected result. > 1 > 2 > 3 > 4 > 5 > 6 > 8 > 8 > 9 > > But you may get different result due to a different value returned by the > rand function in the second run causing different distribution of keys. > This needs to be fixed such that the mapper distributes the same keys even if > it is reattempted multiple times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20220) Incorrect result when hive.groupby.skewindata is enabled
[ https://issues.apache.org/jira/browse/HIVE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591155#comment-16591155 ] Abhishek Somani edited comment on HIVE-20220 at 8/24/18 4:41 AM: - To add to Ganesha's point above, this is an issue with both Tez and MR. So this is a very pertinent issue. was (Author: asomani): To add to Ganesha's point above, this is an issue with Tez as well. So this is a very pertinent issue. > Incorrect result when hive.groupby.skewindata is enabled > > > Key: HIVE-20220 > URL: https://issues.apache.org/jira/browse/HIVE-20220 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 3.0.0 >Reporter: Ganesha Shreedhara >Assignee: Ganesha Shreedhara >Priority: Major > Attachments: HIVE-20220.2.patch, HIVE-20220.patch > > > hive.groupby.skewindata makes use of rand UDF to randomly distribute grouped > by keys to the reducers and hence avoids overloading a single reducer when > there is a skew in data. > This random distribution of keys is buggy when the reducer fails to fetch the > mapper output due to a faulty datanode or any other reason. When reducer > finds that it can't fetch mapper output, it sends a signal to Application > Master to reattempt the corresponding map task. The reattempted map task will > now get the different random value from rand function and hence the keys that > gets distributed now to the reducer will not be same as the previous run. > > *Steps to reproduce:* > create table test(id int); > insert into test values > (1),(2),(2),(3),(3),(3),(4),(4),(4),(4),(5),(5),(5),(5),(5),(6),(6),(6),(6),(6),(6),(7),(7),(7),(7),(7),(7),(7),(7),(8),(8),(8),(8),(8),(8),(8),(8),(9),(9),(9),(9),(9),(9),(9),(9),(9); > SET hive.groupby.skewindata=true; > SET mapreduce.reduce.reduces=2; > //Add a debug port for reducer > select count(1) from test group by id; > //Remove mapper's intermediate output file when map stage is completed and > one out of 2 reduce tasks is completed and then continue the run. This causes > 2nd reducer to send event to Application Master to rerun the map task. > The following is the expected result. > 1 > 2 > 3 > 4 > 5 > 6 > 8 > 8 > 9 > > But you may get different result due to a different value returned by the > rand function in the second run causing different distribution of keys. > This needs to be fixed such that the mapper distributes the same keys even if > it is reattempted multiple times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20220) Incorrect result when hive.groupby.skewindata is enabled
[ https://issues.apache.org/jira/browse/HIVE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606692#comment-16606692 ] Abhishek Somani commented on HIVE-20220: [~gopalv] [~ashutoshc] can someone please help on this one. > Incorrect result when hive.groupby.skewindata is enabled > > > Key: HIVE-20220 > URL: https://issues.apache.org/jira/browse/HIVE-20220 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 3.0.0 >Reporter: Ganesha Shreedhara >Assignee: Ganesha Shreedhara >Priority: Major > Attachments: HIVE-20220.2.patch, HIVE-20220.patch > > > hive.groupby.skewindata makes use of rand UDF to randomly distribute grouped > by keys to the reducers and hence avoids overloading a single reducer when > there is a skew in data. > This random distribution of keys is buggy when the reducer fails to fetch the > mapper output due to a faulty datanode or any other reason. When reducer > finds that it can't fetch mapper output, it sends a signal to Application > Master to reattempt the corresponding map task. The reattempted map task will > now get the different random value from rand function and hence the keys that > gets distributed now to the reducer will not be same as the previous run. > > *Steps to reproduce:* > create table test(id int); > insert into test values > (1),(2),(2),(3),(3),(3),(4),(4),(4),(4),(5),(5),(5),(5),(5),(6),(6),(6),(6),(6),(6),(7),(7),(7),(7),(7),(7),(7),(7),(8),(8),(8),(8),(8),(8),(8),(8),(9),(9),(9),(9),(9),(9),(9),(9),(9); > SET hive.groupby.skewindata=true; > SET mapreduce.reduce.reduces=2; > //Add a debug port for reducer > select count(1) from test group by id; > //Remove mapper's intermediate output file when map stage is completed and > one out of 2 reduce tasks is completed and then continue the run. This causes > 2nd reducer to send event to Application Master to rerun the map task. > The following is the expected result. > 1 > 2 > 3 > 4 > 5 > 6 > 8 > 8 > 9 > > But you may get different result due to a different value returned by the > rand function in the second run causing different distribution of keys. > This needs to be fixed such that the mapper distributes the same keys even if > it is reattempted multiple times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681203#comment-16681203
]
Abhishek Somani commented on HIVE-20823:
[~ekoifman] is this expected to solve
https://issues.apache.org/jira/browse/HIVE-20392 as well?
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809507#comment-16809507
]
Abhishek Somani commented on HIVE-20901:
I think I have a fix for this that I'd like to try and contribute. May I assign
this to myself and submit a patch [~ekoifman]?
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Somani reassigned HIVE-20901:
--
Assignee: Abhishek Somani (was: Eugene Koifman)
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Somani updated HIVE-20901:
---
Attachment: HIVE-20901.1.patch
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Somani updated HIVE-20901:
---
Status: Patch Available (was: Open)
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810472#comment-16810472
]
Abhishek Somani commented on HIVE-20901:
Right. The patch I have uploaded will avoid the duplicate compaction in the
first place.
I also think it is worth fixing this in the 3.1 branch where we do not have
visibility ids.
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Somani updated HIVE-20901:
---
Status: Patch Available (was: Open)
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Somani updated HIVE-20901:
---
Attachment: HIVE-20901.2.patch
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Somani updated HIVE-20901:
---
Status: Open (was: Patch Available)
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16842) insert overwrite with select does not remove data when the select query returns empty resultset
[
https://issues.apache.org/jira/browse/HIVE-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792408#comment-16792408
]
Abhishek Somani commented on HIVE-16842:
[~vgarg] [~ashutoshc] are we planning to fix this?
> insert overwrite with select does not remove data when the select query
> returns empty resultset
> ---
>
> Key: HIVE-16842
> URL: https://issues.apache.org/jira/browse/HIVE-16842
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Priority: Major
>
> {code}create table address (city string);
> insert into address values ('b');
> create table empty_insert (city string);
> insert into empty_insert values ('a');
> insert overwrite table empty_insert select city from address where city='g';
> {code}
> empty_insert still contains 'a'; # should be nothing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-13479) Relax sorting requirement in ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795678#comment-16795678 ] Abhishek Somani commented on HIVE-13479: [~ekoifman] [~vgumashta] [~gopalv] Do we have any plans to work on this? > Relax sorting requirement in ACID tables > > > Key: HIVE-13479 > URL: https://issues.apache.org/jira/browse/HIVE-13479 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Original Estimate: 160h > Remaining Estimate: 160h > > Currently ACID tables require data to be sorted according to internal primary > key. This is that base + delta files can be efficiently sort/merged to > produce the snapshot for current transaction. > This prevents the user to make the table sorted based on any other criteria > which can be useful. One example is using dynamic partition insert (which > also occurs for update/delete SQL). This may create lots of writers > (buckets*partitions) and tax cluster resources. > The usual solution is hive.optimize.sort.dynamic.partition=true which won't > be honored for ACID tables. > We could rely on hash table based algorithm to merge delta files and then not > require any particular sort on Acid tables. One way to do that is to treat > each update event as an Insert (new internal PK) + delete (old PK). Delete > events are very small since they just need to contain PKs. So the hash table > would just need to contain Delete events and be reasonably memory efficient. > This is a significant amount of work but worth doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21165) ACID: pass query hint to the writers to write hive.acid.key.index
[ https://issues.apache.org/jira/browse/HIVE-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795663#comment-16795663 ] Abhishek Somani commented on HIVE-21165: Got it. I was confused between hive.acid.key.index and hive.acid.stats. Is it perhaps then that hive.acid.stats is not required anymore as it just counts types of events? > ACID: pass query hint to the writers to write hive.acid.key.index > - > > Key: HIVE-21165 > URL: https://issues.apache.org/jira/browse/HIVE-21165 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > For the query based compactor from HIVE-20699, the compaction runs as a sql > query. However, this mechanism skips over writing hive.acid.key.index for > each stripe, which is used to skip over stripes that are not supposed to be > read. We need a way to pass a query hint to the writer so that it can write > this index data, when invoked from a sql query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-13479) Relax sorting requirement in ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795678#comment-16795678 ] Abhishek Somani edited comment on HIVE-13479 at 3/19/19 5:01 AM: - [~ekoifman] [~vgumashta] [~gopalv] Do we have any plans to work on this? As I understand it, this might not be a restriction on Insert Only tables as there are no row_ids there, but will need work on Orc readers because of the assumption of records sorted on row_ids for a sort merge of delete events. Is that correct? was (Author: asomani): [~ekoifman] [~vgumashta] [~gopalv] Do we have any plans to work on this? > Relax sorting requirement in ACID tables > > > Key: HIVE-13479 > URL: https://issues.apache.org/jira/browse/HIVE-13479 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Original Estimate: 160h > Remaining Estimate: 160h > > Currently ACID tables require data to be sorted according to internal primary > key. This is that base + delta files can be efficiently sort/merged to > produce the snapshot for current transaction. > This prevents the user to make the table sorted based on any other criteria > which can be useful. One example is using dynamic partition insert (which > also occurs for update/delete SQL). This may create lots of writers > (buckets*partitions) and tax cluster resources. > The usual solution is hive.optimize.sort.dynamic.partition=true which won't > be honored for ACID tables. > We could rely on hash table based algorithm to merge delta files and then not > require any particular sort on Acid tables. One way to do that is to treat > each update event as an Insert (new internal PK) + delete (old PK). Delete > events are very small since they just need to contain PKs. So the hash table > would just need to contain Delete events and be reasonably memory efficient. > This is a significant amount of work but worth doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21165) ACID: pass query hint to the writers to write hive.acid.key.index
[ https://issues.apache.org/jira/browse/HIVE-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794812#comment-16794812 ] Abhishek Somani commented on HIVE-21165: In ACID v2, as every file has only one type of event (insert or delete), isn't hive.acid.key.index not required anymore (as also mentioned in HIVE-20580)? > ACID: pass query hint to the writers to write hive.acid.key.index > - > > Key: HIVE-21165 > URL: https://issues.apache.org/jira/browse/HIVE-21165 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > For the query based compactor from HIVE-20699, the compaction runs as a sql > query. However, this mechanism skips over writing hive.acid.key.index for > each stripe, which is used to skip over stripes that are not supposed to be > read. We need a way to pass a query hint to the writer so that it can write > this index data, when invoked from a sql query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20948) Eliminate file rename in compactor
[
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794814#comment-16794814
]
Abhishek Somani commented on HIVE-20948:
Looks like a duplicate of HIVE-21164.
cc [~vgumashta]
> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have
> compactor write directly to base_x_cZ or delta_x_y_cZ.
> For query based compaction: can we control location of temp table dir? We
> support external temp tables so this may work but we'd need to have non-acid
> insert create files with {{bucket_x}} names.
>
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how
> retries of tasks will work. Just like we currently generate an MR job to
> compact, we should be able to generate a Tez job.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-13479) Relax sorting requirement in ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795933#comment-16795933 ] Abhishek Somani commented on HIVE-13479: What I meant is now that ACID v2 has been implemented, do we plan to work on relaxing the sorting requirement? As far as I know, we still enforce that the rows be sorted on the acid columns(row id), and this is done so that the reader can sort-merge the delete events with the insert events while reading. Isn't that right? If so, it seems the only way to have data sorted on another column specified by the user seems to be to initially insert the data with ordering on that column, so that the data is sorted BOTH on the acid columns as well as user specified column. If however we were able to relax the requirement that data HAS to be sorted on the acid columns, we could utilize something like compaction to sort the data on user desired columns in the background. Theoretically one could do such sorting in compaction even today, but if the sorting requirement is not relaxed, we will need to sort both on row ids and user-column, for which one would need the compaction to behave as an insert overwrite and generate new row ids so that the data is sorted on both the (new)row id columns as well as the user specified column, which would be good to avoid. Have I understood this correct? > Relax sorting requirement in ACID tables > > > Key: HIVE-13479 > URL: https://issues.apache.org/jira/browse/HIVE-13479 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Original Estimate: 160h > Remaining Estimate: 160h > > Currently ACID tables require data to be sorted according to internal primary > key. This is that base + delta files can be efficiently sort/merged to > produce the snapshot for current transaction. > This prevents the user to make the table sorted based on any other criteria > which can be useful. One example is using dynamic partition insert (which > also occurs for update/delete SQL). This may create lots of writers > (buckets*partitions) and tax cluster resources. > The usual solution is hive.optimize.sort.dynamic.partition=true which won't > be honored for ACID tables. > We could rely on hash table based algorithm to merge delta files and then not > require any particular sort on Acid tables. One way to do that is to treat > each update event as an Insert (new internal PK) + delete (old PK). Delete > events are very small since they just need to contain PKs. So the hash table > would just need to contain Delete events and be reasonably memory efficient. > This is a significant amount of work but worth doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765829#comment-16765829
]
Abhishek Somani commented on HIVE-20823:
[~ekoifman], [~vgumashta] Is there a plan to backport this to the 3.x branch?
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch,
> HIVE-20823.08.patch, HIVE-20823.09.patch, HIVE-20823.10.patch,
> HIVE-20823.11.patch, HIVE-20823.11.patch, HIVE-20823.12.patch,
> HIVE-20823.13.patch, HIVE-20823.14.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21058) Make Compactor run in a transaction (Umbrella)
[ https://issues.apache.org/jira/browse/HIVE-21058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765828#comment-16765828 ] Abhishek Somani commented on HIVE-21058: [~ekoifman], [~vgumashta] Is there a plan to backport this to the 3.x branch? > Make Compactor run in a transaction (Umbrella) > -- > > Key: HIVE-21058 > URL: https://issues.apache.org/jira/browse/HIVE-21058 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 4.0.0 > > > Ensure that files produced by the compactor have their visibility controlled > via Hive transaction commit like any other write to an ACID table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814022#comment-16814022
]
Abhishek Somani commented on HIVE-20901:
Looks like HIVE-9995 got merged yesterday with a fix for this, I didn't realize
it was being worked on.
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812969#comment-16812969
]
Abhishek Somani commented on HIVE-20901:
[~ekoifman] [~vgumashta] can you please review the patch? Thanks!
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21714) INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting with ACID table
[ https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838245#comment-16838245 ] Abhishek Somani commented on HIVE-21714: [~isuller] sorry I might have misunderstood, but isn't it expected that an insert overwrite of an acid table will retain old data? Is this Jira about cleaning up old data of an acid table when we do an insert overwrite? > INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting > with ACID table > --- > > Key: HIVE-21714 > URL: https://issues.apache.org/jira/browse/HIVE-21714 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch > > > The issue of HIVE-18702 is present for ACID tables as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21714) INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting with ACID table
[ https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838293#comment-16838293 ] Abhishek Somani commented on HIVE-21714: You can see HIVE-14988. As in the description, old data will be retained and concurrent readers could be allowed to read while an IOW is happening. "The advantage is that IOW doesn't need to be a mutexed operation. All readers that stared before this IOW can proceed at the snapshot (Acid tables support at Snapshot Isolation) they locked in." Although this doesn't happen currently as an IOW takes an exclusive lock(X), I think the design is intended this way so that the lock that IOW takes could be relaxed in the future(and perhaps allow concurrent readers). > INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting > with ACID table > --- > > Key: HIVE-21714 > URL: https://issues.apache.org/jira/browse/HIVE-21714 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch > > > The issue of HIVE-18702 is present for ACID tables as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21714) Insert overwrite on an acid/mm table is ineffective if the input is empty
[ https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838331#comment-16838331 ] Abhishek Somani commented on HIVE-21714: Got it! Thanks. > Insert overwrite on an acid/mm table is ineffective if the input is empty > - > > Key: HIVE-21714 > URL: https://issues.apache.org/jira/browse/HIVE-21714 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch > > > The issue of HIVE-18702 is present for ACID tables as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21513) ACID: Running merge concurrently with minor compaction causes a later select * to throw exception
[ https://issues.apache.org/jira/browse/HIVE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857265#comment-16857265 ] Abhishek Somani commented on HIVE-21513: Can you please tell [~vgumashta] which Jira is this a duplicate of? > ACID: Running merge concurrently with minor compaction causes a later select > * to throw exception > -- > > Key: HIVE-21513 > URL: https://issues.apache.org/jira/browse/HIVE-21513 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0, 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > > Repro steps: > - Create table > - Load some data > - Run merge so records gets updated and delete_delta dirs are created > - Manually initiate minor compaction: ALTER TABLE ... COMPACT 'minor'; > - While the compaction is running keep executing the merge statement > - After some time try to do simple select *; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21513) ACID: Running merge concurrently with minor compaction causes a later select * to throw exception
[ https://issues.apache.org/jira/browse/HIVE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857265#comment-16857265 ] Abhishek Somani edited comment on HIVE-21513 at 6/6/19 4:04 AM: [~vgumashta] Can you please tell which Jira is this a duplicate of? was (Author: asomani): Can you please tell [~vgumashta] which Jira is this a duplicate of? > ACID: Running merge concurrently with minor compaction causes a later select > * to throw exception > -- > > Key: HIVE-21513 > URL: https://issues.apache.org/jira/browse/HIVE-21513 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0, 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > > Repro steps: > - Create table > - Load some data > - Run merge so records gets updated and delete_delta dirs are created > - Manually initiate minor compaction: ALTER TABLE ... COMPACT 'minor'; > - While the compaction is running keep executing the merge statement > - After some time try to do simple select *; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21841) Leader election in HMS to run housekeeping tasks.
[ https://issues.apache.org/jira/browse/HIVE-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863835#comment-16863835 ] Abhishek Somani commented on HIVE-21841: [~ashutosh.bapat] as I understand it currently, multiple HMS's co-ordinate amongst each other and ensure that only one of them is doing the job at a time by locking an entry in the db, in the AUX_TABLE. We think that its not good enough? > Leader election in HMS to run housekeeping tasks. > - > > Key: HIVE-21841 > URL: https://issues.apache.org/jira/browse/HIVE-21841 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21841.01.patch, HIVE-21841.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HMS performs housekeeping tasks. When there are multiple HMSes we need to > have a leader HMS elected which will carry out those housekeeping tasks. > These tasks include execution of compaction tasks, auto-discovering > partitions for external tables, generation of compaction tasks, repl thread > etc. > Note that, though the code for compaction tasks, auto-discovery of partitions > etc. is in Hive, the actual tasks are initiated by an HMS configured to do > so. So, leader election is required only for HMS and not for HS2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-22413) Avoid dirty read when reading the ACID table while compaction is running
[ https://issues.apache.org/jira/browse/HIVE-22413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962817#comment-16962817 ] Abhishek Somani commented on HIVE-22413: [~pvary] an issue with HIVE-20823 is that it is in 4.0.0(master) only. Backporting it to Hive 2/Hive 3 is not feasible as it is a major design change. I think we need an interim solution for S3/other blobstores in older Hive versions. We solved this in a different way ourselves. At the end of compaction, we insert a \_compaction_done file in the compacted directory, and the readers have been modified (in getAcidState()) to ignore base/delta directories till this file is visible. > Avoid dirty read when reading the ACID table while compaction is running > > > Key: HIVE-22413 > URL: https://issues.apache.org/jira/browse/HIVE-22413 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Hocheol Park >Priority: Major > Attachments: HIVE-22413.1.patch > > > There is a problem that dirty read occurs when reading the ACID table while > base or delta directories are being created by the compactor. Especially it > is highly likely to occur in the S3 storage because the “move” logic of S3 is > “copy and delete”, and it takes a long time to copy if the size of files are > large or bucketing count is large. > So here’s the logic to avoid this problem. If “_tmp” prefixed directories are > existed in the partition directory on the process of listing the child > directories when reading the ACID table, compare the names of the directory > in the “_tmp” one and skip it in case of the same. Then it will read the > files before merging, no difference on the results. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20901) running compactor when there is nothing to do produces duplicate data
[
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010587#comment-17010587
]
Abhishek Somani commented on HIVE-20901:
[~pvary] Sure!
> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think
> there is a check for that. It's visible in the context of HIVE-20823, where
> each compactor run produces a delta with new visibility suffix so we end up
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delete_delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_001_
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v019
> │ ├── _orc_acid_version
> │ └── bucket_0
> ├── delta_001_002_v021
> │ ├── _orc_acid_version
> │ └── bucket_0
> └── delta_002_002_
> ├── _orc_acid_version
> └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad. Probably happens today as well but new run produces a delta
> with the same name and clobbers the previous one, which may interfere with
> writers
>
> need to investigate
>
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both
> deltas as if they were distinct and it effectively duplicates data.- There
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the
> same {{writeid}} range
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23143) Transactions: PPD in Delete deltas is broken
[
https://issues.apache.org/jira/browse/HIVE-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076404#comment-17076404
]
Abhishek Somani commented on HIVE-23143:
[~pvary] it doesn't help. I tested on current master branch (as of last week)
that has the HIVE-22880 patcg.
It doesn't help presumably because what HIVE-22880 is saying is ignore all
SARGs for *data* columns. But the issue is *metadata* column SARGs (ie like on
transactionids, bucket etc) when applied, are applied incorrectly.
> Transactions: PPD in Delete deltas is broken
>
>
> Key: HIVE-23143
> URL: https://issues.apache.org/jira/browse/HIVE-23143
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Abhishek Somani
>Priority: Major
>
> The optimization introduced in HIVE-16812 seems broken. PPD is not happening
> for delete deltas, and in fact, also causes wrong results if data column
> names conflict with ACID ROW__ID column names (bucket, originalTransactionId
> etc).
> This seems to be happening because after ORC-491, all PPD happens in data
> columns only for ACID orc files, so the filters for delete PPD never get
> applied on metadata columns and try to apply to data columns instead. And
> when the data columns have a column name (like "bucket" in the below
> example), it returns wrong results.
> Steps to repro:
> {code:java}
> set hive.fetch.task.conversion=none;
> set hive.query.results.cache.enabled=false;
> create table test(a int, bucket int) stored as orc
> tblproperties("transactional"="true");
> insert into table test values (1, ), (2, ), (3, );
> delete from test where a = 2;
> select * from test; //Will return the deleted row as well
> set hive.txn.filter.delete.events=false;
> select * from test; //Correct results returned. Will not return the deleted
> row
> {code}
> cc [~pvary] [~gopalv]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23143) Transactions: PPD in Delete deltas is broken
[
https://issues.apache.org/jira/browse/HIVE-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076404#comment-17076404
]
Abhishek Somani edited comment on HIVE-23143 at 4/6/20, 3:21 PM:
-
[~pvary] it doesn't help. I tested on current master branch (as of last week)
that has the HIVE-22880 patch.
It doesn't help presumably because what HIVE-22880 is saying is ignore all
SARGs for *data* columns. But the issue is *metadata* column SARGs (ie like on
transactionids, bucket etc) when applied (even after ignoring *data* column
sargs), are applied incorrectly.
was (Author: asomani):
[~pvary] it doesn't help. I tested on current master branch (as of last week)
that has the HIVE-22880 patch.
It doesn't help presumably because what HIVE-22880 is saying is ignore all
SARGs for *data* columns. But the issue is *metadata* column SARGs (ie like on
transactionids, bucket etc) when applied, are applied incorrectly.
> Transactions: PPD in Delete deltas is broken
>
>
> Key: HIVE-23143
> URL: https://issues.apache.org/jira/browse/HIVE-23143
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Abhishek Somani
>Priority: Major
>
> The optimization introduced in HIVE-16812 seems broken. PPD is not happening
> for delete deltas, and in fact, also causes wrong results if data column
> names conflict with ACID ROW__ID column names (bucket, originalTransactionId
> etc).
> This seems to be happening because after ORC-491, all PPD happens in data
> columns only for ACID orc files, so the filters for delete PPD never get
> applied on metadata columns and try to apply to data columns instead. And
> when the data columns have a column name (like "bucket" in the below
> example), it returns wrong results.
> Steps to repro:
> {code:java}
> set hive.fetch.task.conversion=none;
> set hive.query.results.cache.enabled=false;
> create table test(a int, bucket int) stored as orc
> tblproperties("transactional"="true");
> insert into table test values (1, ), (2, ), (3, );
> delete from test where a = 2;
> select * from test; //Will return the deleted row as well
> set hive.txn.filter.delete.events=false;
> select * from test; //Correct results returned. Will not return the deleted
> row
> {code}
> cc [~pvary] [~gopalv]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23143) Transactions: PPD in Delete deltas is broken
[
https://issues.apache.org/jira/browse/HIVE-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076404#comment-17076404
]
Abhishek Somani edited comment on HIVE-23143 at 4/6/20, 3:21 PM:
-
[~pvary] it doesn't help. I tested on current master branch (as of last week)
that has the HIVE-22880 patch.
It doesn't help presumably because what HIVE-22880 is saying is ignore all
SARGs for *data* columns. But the issue is *metadata* column SARGs (ie like on
transactionids, bucket etc) when applied, are applied incorrectly.
was (Author: asomani):
[~pvary] it doesn't help. I tested on current master branch (as of last week)
that has the HIVE-22880 patcg.
It doesn't help presumably because what HIVE-22880 is saying is ignore all
SARGs for *data* columns. But the issue is *metadata* column SARGs (ie like on
transactionids, bucket etc) when applied, are applied incorrectly.
> Transactions: PPD in Delete deltas is broken
>
>
> Key: HIVE-23143
> URL: https://issues.apache.org/jira/browse/HIVE-23143
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Abhishek Somani
>Priority: Major
>
> The optimization introduced in HIVE-16812 seems broken. PPD is not happening
> for delete deltas, and in fact, also causes wrong results if data column
> names conflict with ACID ROW__ID column names (bucket, originalTransactionId
> etc).
> This seems to be happening because after ORC-491, all PPD happens in data
> columns only for ACID orc files, so the filters for delete PPD never get
> applied on metadata columns and try to apply to data columns instead. And
> when the data columns have a column name (like "bucket" in the below
> example), it returns wrong results.
> Steps to repro:
> {code:java}
> set hive.fetch.task.conversion=none;
> set hive.query.results.cache.enabled=false;
> create table test(a int, bucket int) stored as orc
> tblproperties("transactional"="true");
> insert into table test values (1, ), (2, ), (3, );
> delete from test where a = 2;
> select * from test; //Will return the deleted row as well
> set hive.txn.filter.delete.events=false;
> select * from test; //Correct results returned. Will not return the deleted
> row
> {code}
> cc [~pvary] [~gopalv]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088800#comment-17088800 ] Abhishek Somani commented on HIVE-8123: --- [~gopalv] [~pvary] [~omalley] Can you please take a look at the above design doc. We have made some progress and will upload a patch soon as well, working on test cases right now. > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105622#comment-17105622 ] Abhishek Somani commented on HIVE-21052: This seems like a critical issue. Is this actively being worked on? This affects both Hive 3 and current master (Hive 4), right? cc [~pvary] [~gopalv] > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0, 3.1.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, > HIVE-21052.8.patch, HIVE-21052.9.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107480#comment-17107480 ] Abhishek Somani commented on HIVE-21052: Yes [~pvary] we are taking a look at this and will get back. > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0, 3.1.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, > HIVE-21052.8.patch, HIVE-21052.9.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23804) Adding defaults for Columns Stats table in the schema to make them backward compatible
[ https://issues.apache.org/jira/browse/HIVE-23804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158515#comment-17158515 ] Abhishek Somani commented on HIVE-23804: [~ngangam] the issue exists in all metastore schemas of 3.0 and above. When any such schema is used with an HMS version <= 3.0, one will hit the issue. So for example, if you use metastore db schema of 4.0.0, but HMS version 2.3, one will hit the issue. So basically the backward compatibility of metastore db with older HMS versions is broken. > Adding defaults for Columns Stats table in the schema to make them backward > compatible > -- > > Key: HIVE-23804 > URL: https://issues.apache.org/jira/browse/HIVE-23804 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.1, 2.3.7 >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-23804-1.patch, HIVE-23804.patch > > > Since the table/part column statistics tables have added a new `CAT_NAME` > column with `NOT NULL` constraint in version >3.0.0, queries to analyze > statistics break for Hive versions <3.0.0 when used against an upgraded DB. > One such miss is handled in HIVE-21739. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23804) Adding defaults for Columns Stats table in the schema to make them backward compatible
[ https://issues.apache.org/jira/browse/HIVE-23804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158515#comment-17158515 ] Abhishek Somani edited comment on HIVE-23804 at 7/15/20, 4:33 PM: -- [~ngangam] the issue exists in all metastore schemas of 3.0 and above. When any such schema is used with an HMS version < 3.0, one will hit the issue. So for example, if you use metastore db schema of 4.0.0, but HMS version 2.3, one will hit the issue. So basically the backward compatibility of metastore db with older HMS versions is broken. was (Author: asomani): [~ngangam] the issue exists in all metastore schemas of 3.0 and above. When any such schema is used with an HMS version <= 3.0, one will hit the issue. So for example, if you use metastore db schema of 4.0.0, but HMS version 2.3, one will hit the issue. So basically the backward compatibility of metastore db with older HMS versions is broken. > Adding defaults for Columns Stats table in the schema to make them backward > compatible > -- > > Key: HIVE-23804 > URL: https://issues.apache.org/jira/browse/HIVE-23804 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.1, 2.3.7 >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-23804-1.patch, HIVE-23804.patch > > > Since the table/part column statistics tables have added a new `CAT_NAME` > column with `NOT NULL` constraint in version >3.0.0, queries to analyze > statistics break for Hive versions <3.0.0 when used against an upgraded DB. > One such miss is handled in HIVE-21739. -- This message was sent by Atlassian Jira (v8.3.4#803005)
