[
https://issues.apache.org/jira/browse/IMPALA-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-12812:
-------------------------------------
Description:
IMPALA-11808 added support for sending reload events after REFRESH to allow
other Impala cluster connecting to the same HMS to also reload their tables.
REFRESH is often used when in external tables the files are written directly to
filesystem without notifying HMS, so Impala needs to update its cache and can't
rely on HMS notifications.
The same could be useful for ALTER TABLE RECOVER PARTITIONS. {-}It detects
partition directories that were only created in the FS but not in HMS and
creates them in HMS too. I{-}t also reloads the table (similarly to other DDLs)
and detects new files in existing partitions. - UPDATE: the previous sentence
was not true with current Impala.
An HMS event is created for the new partitions but there is no event that would
indicate that there are new files in existing partitions. As ALTER TABLE
RECOVER PARTITIONS is called when the user expects changes in the filesystem
(similarly to REFRESH), it could be useful to send a reload event after it is
finished.
was:
IMPALA-11808 added support for sending reload events after REFRESH to allow
other Impala cluster connecting to the same HMS to also reload their tables.
REFRESH is often used when in external tables the files are written directly to
filesystem without notifying HMS, so Impala needs to update its cache and can't
rely on HMS notifications.
The same could be useful for ALTER TABLE RECOVER PARTITIONS. {-}- It detects
partition directories that were only created in the FS but not in HMS and
creates them in HMS too.-{-}It also reloads the table (similarly to other DDLs)
and detects new files in existing partitions. - UPDATE: the previous sentence
was not true with current Impala.
An HMS event is created for the new partitions but there is no event that would
indicate that there are new files in existing partitions. As ALTER TABLE
RECOVER PARTITIONS is called when the user expects changes in the filesystem
(similarly to REFRESH), it could be useful to send a reload event after it is
finished.
> Send reload event after ALTER TABLE RECOVER PARTITIONS
> ------------------------------------------------------
>
> Key: IMPALA-12812
> URL: https://issues.apache.org/jira/browse/IMPALA-12812
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Csaba Ringhofer
> Priority: Major
>
> IMPALA-11808 added support for sending reload events after REFRESH to allow
> other Impala cluster connecting to the same HMS to also reload their tables.
> REFRESH is often used when in external tables the files are written directly
> to filesystem without notifying HMS, so Impala needs to update its cache and
> can't rely on HMS notifications.
> The same could be useful for ALTER TABLE RECOVER PARTITIONS. {-}It detects
> partition directories that were only created in the FS but not in HMS and
> creates them in HMS too. I{-}t also reloads the table (similarly to other
> DDLs) and detects new files in existing partitions. - UPDATE: the previous
> sentence was not true with current Impala.
> An HMS event is created for the new partitions but there is no event that
> would indicate that there are new files in existing partitions. As ALTER
> TABLE RECOVER PARTITIONS is called when the user expects changes in the
> filesystem (similarly to REFRESH), it could be useful to send a reload event
> after it is finished.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
