[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Fix Version/s: 4.0.0-alpha-1
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the presence
> of open txn (by `ALTER TABLE`) below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
> AcidUtils.processBaseDir
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
> validTxnList)) {
>return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-23832:
--
Labels: pull-request-available (was: )
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the presence
> of open txn (by `ALTER TABLE`) below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
> AcidUtils.processBaseDir
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
> validTxnList)) {
>return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Description:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the presence of
open txn (by `ALTER TABLE`) below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
AcidUtils.processBaseDir
{code}
if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
validTxnList)) {
return;
}
{code}
was:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn by
`ALTER TABLE` below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
AcidUtils.processBaseDir
{code}
if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
validTxnList)) {
return;
}
{code}
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the presence
> of open txn (by `ALTER TABLE`) below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
> AcidUtils.processBaseDir
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
> validTxnList)) {
>return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Description:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn by
`ALTER TABLE` below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
{code}
if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
validTxnList)) {
return;
}
{code}
was:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn by
`ALTER TABLE` below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
{code}
if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
validTxnList)) {
return;
}
{code}
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the open txn
> by `ALTER TABLE` below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
> validTxnList)) {
>return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Description:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn by
`ALTER TABLE` below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
{code}
if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
validTxnList)) {
return;
}
{code}
was:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn (by
ALTER TABLE) below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the open txn
> by `ALTER TABLE` below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
> validTxnList)) {
> return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Description:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn by
`ALTER TABLE` below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
AcidUtils.processBaseDir
{code}
if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
validTxnList)) {
return;
}
{code}
was:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn by
`ALTER TABLE` below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
{code}
if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
validTxnList)) {
return;
}
{code}
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the open txn
> by `ALTER TABLE` below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
> AcidUtils.processBaseDir
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
> validTxnList)) {
>return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Description:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn (by
ALTER TABLE) below Compactor's one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
was:
{code}
CREATE TABLE default.compcleanup (
cda_id int,
cda_run_id varchar(255),
cda_load_tstimestamp,
global_party_idstring,
group_id string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
cda_date int,
cda_job_name varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES
(1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name =
'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
{code}
When using blocking compaction Cleaner skips processing due to the open txn (by
ALTER TABLE) below Compactor one.
{code}
AcidUtils - getChildState() ignoring([])
pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
{code}
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the open txn
> (by ALTER TABLE) below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Summary: Compaction cleaner fails to clean up deltas when using blocking
compaction (was: Cleaner skips processing when using blocking compaction)
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the open txn
> (by ALTER TABLE) below Compactor one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
