[
https://issues.apache.org/jira/browse/FLINK-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863503#comment-16863503
]
Xuefu Zhang commented on FLINK-12771:
-------------------------------------
> [~bdaw] commented on PR #8720:
> I don't think this feature makes any sense. This makes the HiveCatalog also
> an instance of InMemoryCatalog.
>
> I think if a table cannot be converted to properties (as is the case for
> ConnectorCatalogTable an exception should be thrown. We
> should allow storing those kind of tables only in an in memory catalog.
Hi [~bdaw], thanks for sharing your concern. Other than the fact that there is
some overlap with an in-memory catalog, could you elaborate why the feature
doesn't make sense?
In my humble opinion, this feature makes many senses:
1. Catalog API defined an API that make an implementation expect to handle this
type of table. The implementation can choose to how to handle it. Throwing an
exception in one implementation is one of the options, but it's also reasonable
that another one to handle differently.
2. {{ConnectorCatalogTable}} is temporary in nature, and Hive catalog supports
temporary objects (tables, functions). Supporting {{ConnectorCatalogTable}} in
{{HiveCatalog}} is a natural fit.
3. Not being able to serialize {{ConnectorCatalogTable}} shouldn't be a reason
why it should be rejected by {{HiveCatalog}}. Serialization is possible if we
try hard enough. However, the nature of the table requires no serialization.
4. Supporting {{ConnectorCatalogTable}} brings better usability as user doesn't
have to register two catalogs and juggling between them. The JIRA description
has particularly stressed on this.
5. A catalog is not limited in supporting just one type of tables.
{{HiveCatalog}} already supports both generic tables and native Hive tables.
With this feature, it also supports inline tables.
The only downside as I see is the introduced complexity. Again this is just
implementation details and a user shouldn't care.
> Support ConnectorCatalogTable in HiveCatalog
> --------------------------------------------
>
> Key: FLINK-12771
> URL: https://issues.apache.org/jira/browse/FLINK-12771
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{HiveCatalog}} does not support {{ConnectorCatalogTable}}. There's
> a major drawback on this when it comes to real use cases, that is when Table
> API users set a {{HiveCatalog}} as their default catalog (which is very
> likely), they cannot create or use any inline table sources/sinks with their
> default catalog any more. It's really inconvenient for Table API users to use
> Flink for exploration, experiment, and production.
> There are several workaround in this case. E.g. users have to switch their
> default catalog, but that misses our original intention of having a default
> {{HiveCatalog}}; or users can register their inline source/sinks to Flink's
> default catalog which is a in memory catalog, but that not only require users
> to type full path of a table but also requires users to be aware of the
> Flink's default catalog, default db, and their names. In short, none of the
> workaround seems to be reasonable and user friendly.
> From another perspective, Hive has the concept of temporary tables that are
> stored in memory of Hive metastore client and are removed when client is shut
> down. In Flink, {{ConnectorCatalogTable}} can be seen as a type of
> session-based temporary table, and {{HiveCatalog}} (potentially any catalog
> implementations) can store it in memory. By introducing the concept of temp
> table, we could greatly eliminate frictions for users and raise their
> experience and productivity.
> Thus, we propose adding a simple in memory map for {{ConnectorCatalogTable}}
> in {{HiveCatalog}} to allow users create and use inline source/sink when
> their default catalog is a {{HiveCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
