[
https://issues.apache.org/jira/browse/IMPALA-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822631#comment-17822631
]
Csaba Ringhofer edited comment on IMPALA-12812 at 3/1/24 4:14 PM:
------------------------------------------------------------------
I was wrong about this one:
" It also reloads the table (similarly to other DDLs) and detects new files in
existing partitions. "
At the moment no refresh is done on partitions that already exist in HMS.
A valid workaround is to call both REFRESH after ALTER TABLE RECOVER PARTITIONS
- REFRESH will both detect new files and send the reload event.
Closing the issue as it wouldn't be that useful.
was (Author: csringhofer):
I was wrong about this one:
"An HMS event is created for the new partitions but there is no event that
would indicate that there are new files in existing partitions. "
At the moment no refresh is done on partitions that already exist in HMS.
A valid workaround is to call both REFRESH after ALTER TABLE RECOVER PARTITIONS
- REFRESH will both detect new files and send the reload event.
Closing the issue as it wouldn't be that useful.
> Send reload event after ALTER TABLE RECOVER PARTITIONS
> ------------------------------------------------------
>
> Key: IMPALA-12812
> URL: https://issues.apache.org/jira/browse/IMPALA-12812
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Csaba Ringhofer
> Priority: Major
>
> IMPALA-11808 added support for sending reload events after REFRESH to allow
> other Impala cluster connecting to the same HMS to also reload their tables.
> REFRESH is often used when in external tables the files are written directly
> to filesystem without notifying HMS, so Impala needs to update its cache and
> can't rely on HMS notifications.
> The same could be useful for ALTER TABLE RECOVER PARTITIONS. It detects
> partition directories that were only created in the FS but not in HMS and
> creates them in HMS too. It also reloads the table (similarly to other DDLs)
> and detects new files in existing partitions. An HMS event is created for the
> new partitions but there is no event that would indicate that there are new
> files in existing partitions. As ALTER TABLE RECOVER PARTITIONS is called
> when the user expects changes in the filesystem (similarly to REFRESH), it
> could be useful to send a reload event after it is finished.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
