[jira] [Updated] (HIVE-15920) Implement a blocking version of a command to compact
[
https://issues.apache.org/jira/browse/HIVE-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-15920:
--
Labels: TODOC2.2 (was: )
> Implement a blocking version of a command to compact
>
>
> Key: HIVE-15920
> URL: https://issues.apache.org/jira/browse/HIVE-15920
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15920.01.patch, HIVE-15920.02.patch
>
>
> currently
> {noformat}
> alter table AcidTable compact 'major'
> {noformat}
> is supported which enqueues a msg to compact.
> Would be nice for testing and script building to support
> {noformat}
> alter table AcidTable compact 'major' blocking
> {noformat}
> perhaps another variation is to block until either compaction is done or
> until cleaning is finished.
> DDLTask.compact() gets a request id back so it can then just block and wait
> for it using some new API
> may also be useful to let users compact all partitions but only if a
> separate queue has been set up for compaction jobs.
> The later is because with a 1M partition table, this may create very many
> jobs and saturate the cluster.
> This probably requires HIVE-12376 to make sure the compaction queue does the
> throttling, not the number of worker threads
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15920) Implement a blocking version of a command to compact
[
https://issues.apache.org/jira/browse/HIVE-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15920:
--
committed to master
thanks Wei for the review
> Implement a blocking version of a command to compact
>
>
> Key: HIVE-15920
> URL: https://issues.apache.org/jira/browse/HIVE-15920
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 2.2.0
>
> Attachments: HIVE-15920.01.patch, HIVE-15920.02.patch
>
>
> currently
> {noformat}
> alter table AcidTable compact 'major'
> {noformat}
> is supported which enqueues a msg to compact.
> Would be nice for testing and script building to support
> {noformat}
> alter table AcidTable compact 'major' blocking
> {noformat}
> perhaps another variation is to block until either compaction is done or
> until cleaning is finished.
> DDLTask.compact() gets a request id back so it can then just block and wait
> for it using some new API
> may also be useful to let users compact all partitions but only if a
> separate queue has been set up for compaction jobs.
> The later is because with a 1M partition table, this may create very many
> jobs and saturate the cluster.
> This probably requires HIVE-12376 to make sure the compaction queue does the
> throttling, not the number of worker threads
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15920) Implement a blocking version of a command to compact
[
https://issues.apache.org/jira/browse/HIVE-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15920:
--
Attachment: HIVE-15920.02.patch
> Implement a blocking version of a command to compact
>
>
> Key: HIVE-15920
> URL: https://issues.apache.org/jira/browse/HIVE-15920
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15920.01.patch, HIVE-15920.02.patch
>
>
> currently
> {noformat}
> alter table AcidTable compact 'major'
> {noformat}
> is supported which enqueues a msg to compact.
> Would be nice for testing and script building to support
> {noformat}
> alter table AcidTable compact 'major' blocking
> {noformat}
> perhaps another variation is to block until either compaction is done or
> until cleaning is finished.
> DDLTask.compact() gets a request id back so it can then just block and wait
> for it using some new API
> may also be useful to let users compact all partitions but only if a
> separate queue has been set up for compaction jobs.
> The later is because with a 1M partition table, this may create very many
> jobs and saturate the cluster.
> This probably requires HIVE-12376 to make sure the compaction queue does the
> throttling, not the number of worker threads
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15920) Implement a blocking version of a command to compact
[
https://issues.apache.org/jira/browse/HIVE-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15920:
--
Attachment: HIVE-15920.01.patch
> Implement a blocking version of a command to compact
>
>
> Key: HIVE-15920
> URL: https://issues.apache.org/jira/browse/HIVE-15920
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15920.01.patch
>
>
> currently
> {noformat}
> alter table AcidTable compact 'major'
> {noformat}
> is supported which enqueues a msg to compact.
> Would be nice for testing and script building to support
> {noformat}
> alter table AcidTable compact 'major' blocking
> {noformat}
> perhaps another variation is to block until either compaction is done or
> until cleaning is finished.
> DDLTask.compact() gets a request id back so it can then just block and wait
> for it using some new API
> may also be useful to let users compact all partitions but only if a
> separate queue has been set up for compaction jobs.
> The later is because with a 1M partition table, this may create very many
> jobs and saturate the cluster.
> This probably requires HIVE-12376 to make sure the compaction queue does the
> throttling, not the number of worker threads
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15920) Implement a blocking version of a command to compact
[
https://issues.apache.org/jira/browse/HIVE-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15920:
--
Status: Patch Available (was: Open)
> Implement a blocking version of a command to compact
>
>
> Key: HIVE-15920
> URL: https://issues.apache.org/jira/browse/HIVE-15920
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15920.01.patch
>
>
> currently
> {noformat}
> alter table AcidTable compact 'major'
> {noformat}
> is supported which enqueues a msg to compact.
> Would be nice for testing and script building to support
> {noformat}
> alter table AcidTable compact 'major' blocking
> {noformat}
> perhaps another variation is to block until either compaction is done or
> until cleaning is finished.
> DDLTask.compact() gets a request id back so it can then just block and wait
> for it using some new API
> may also be useful to let users compact all partitions but only if a
> separate queue has been set up for compaction jobs.
> The later is because with a 1M partition table, this may create very many
> jobs and saturate the cluster.
> This probably requires HIVE-12376 to make sure the compaction queue does the
> throttling, not the number of worker threads
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15920) Implement a blocking version of a command to compact
[
https://issues.apache.org/jira/browse/HIVE-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15920:
--
Description:
currently
{noformat}
alter table AcidTable compact 'major'
{noformat}
is supported which enqueues a msg to compact.
Would be nice for testing and script building to support
{noformat}
alter table AcidTable compact 'major' blocking
{noformat}
perhaps another variation is to block until either compaction is done or until
cleaning is finished.
DDLTask.compact() gets a request id back so it can then just block and wait for
it using some new API
may also be useful to let users compact all partitions but only if a separate
queue has been set up for compaction jobs.
The later is because with a 1M partition table, this may create very many jobs
and saturate the cluster.
This probably requires HIVE-12376 to make sure the compaction queue does the
throttling, not the number of worker threads
was:
currently
{noformat}
alter table AcidTable compact 'major'
{noformat}
is supported which enqueues a msg to compact.
Would be nice for testing and script building to support
{noformat}
alter table AcidTable compact 'major' blocking
{noformat}
perhaps another variation is to block until either compaction is done or until
cleaning is finished.
DDLTask.compact() gets a request id back so it can then just block and wait for
it using some new API
> Implement a blocking version of a command to compact
>
>
> Key: HIVE-15920
> URL: https://issues.apache.org/jira/browse/HIVE-15920
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> currently
> {noformat}
> alter table AcidTable compact 'major'
> {noformat}
> is supported which enqueues a msg to compact.
> Would be nice for testing and script building to support
> {noformat}
> alter table AcidTable compact 'major' blocking
> {noformat}
> perhaps another variation is to block until either compaction is done or
> until cleaning is finished.
> DDLTask.compact() gets a request id back so it can then just block and wait
> for it using some new API
> may also be useful to let users compact all partitions but only if a
> separate queue has been set up for compaction jobs.
> The later is because with a 1M partition table, this may create very many
> jobs and saturate the cluster.
> This probably requires HIVE-12376 to make sure the compaction queue does the
> throttling, not the number of worker threads
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15920) Implement a blocking version of a command to compact
[
https://issues.apache.org/jira/browse/HIVE-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15920:
--
Description:
currently
{noformat}
alter table AcidTable compact 'major'
{noformat}
is supported which enqueues a msg to compact.
Would be nice for testing and script building to support
{noformat}
alter table AcidTable compact 'major' blocking
{noformat}
perhaps another variation is to block until either compaction is done or until
cleaning is finished.
DDLTask.compact() gets a request id back so it can then just block and wait for
it using some new API
was:
currently
{noformat}
alter table AcidTable compact 'major'
{noformat}
is supported which enqueues a msg to compact.
Would be nice for testing and script building to support
{noformat}
alter table AcidTable compact 'major' blocking
{noformat}
perhaps another variation is to block until either compaction is done or until
cleaning is finished.
> Implement a blocking version of a command to compact
>
>
> Key: HIVE-15920
> URL: https://issues.apache.org/jira/browse/HIVE-15920
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> currently
> {noformat}
> alter table AcidTable compact 'major'
> {noformat}
> is supported which enqueues a msg to compact.
> Would be nice for testing and script building to support
> {noformat}
> alter table AcidTable compact 'major' blocking
> {noformat}
> perhaps another variation is to block until either compaction is done or
> until cleaning is finished.
> DDLTask.compact() gets a request id back so it can then just block and wait
> for it using some new API
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
