[
https://issues.apache.org/jira/browse/KYLIN-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kun Liu updated KYLIN-4921:
---------------------------
Description:
In project A, I have created a stream table test_table.
When I create the table test_table in project B, the table can't be created,
but the source config of table in project A lost.
The reason:
In the `saveStreamingConfig` function, if creating table failed, the config
will be dropped in finally phase.
By design:
In order to identity one stream table `StreamingSourceConfig` in the metastore,
kylin use the key
`ResourceStore.STREAMING_V2_RESOURCE_ROOT + "/" + name + ".json"` as the
identifier.
But this design can't work in multi-project.
[~hit_lacus] PTAL
[Design
Doc|https://docs.google.com/document/u/1/d/1QfTn5dBcHeY2EAnMUS2V0sqj9dpFQwmVuAZon7wCtns/edit#]
Design:
h1. The Rowkey of the Stream source config
h1. Origin design
In the project `project_test`, we create a real-time stream table with the name
of `stream_table`, which will create two metadata, one the `tableDesc` and
another is `streamSourceConfig`.
The `tableDesc` is stored in the hbase by the path
`table_prefix/tablename--projectname.json`, but the `streamSourceConfig` is
stored in the hbase by the path `stream_source_prefix/tablename.json`. The
action of creating tables with the same name in different projects is not
allowed.
h1. New design
The new rowkey for storing the `streamSourceConfig` is
`stream_source_prefix/tablename--projectname.json`.
h1. How to deal with the compatibility in kylin
h2. The type of RowKey
* NewRowKey: `stream_source_prefix/tablename--projectname.json`
* OldRowKey: `stream_source_prefix/tablename.json`
h2. The type of operation
* save source
* update source
* query/get source
* delete
h3. saveStreamingConfig:
* store the stream source config with NewRowKey
h3. removeStreamingConfig:
* if the source config exist in the NewRowKey, delete the source config using
the NewRowKey
* if the source config exist in the OldRowKey, delete the source config using
the OldRowKey
h3. updateStreamingConfig:
* removeStreamingConfig
* saveStreamingConfig
h3. reloadStreamingConfigLocal/queryStreamingConfig
Hypothesis: The stream config must exist.
* check NewRowKey, if the source config exists, return the object.
* if the source config doesn't exist, and the source config exists in the
OldRowKey
* get the Source config and update the project name of the source config
* delete the source config in the OldRowKey and resave the source config in
the NewRowKey
was:
In project A, I have created a stream table test_table.
When I create the table test_table in project B, the table can't be created,
but the source config of table in project A lost.
The reason:
In the `saveStreamingConfig` function, if creating table failed, the config
will be dropped in finally phase.
By design:
In order to identity one stream table `StreamingSourceConfig` in the metastore,
kylin use the key
`ResourceStore.STREAMING_V2_RESOURCE_ROOT + "/" + name + ".json"` as the
identifier.
But this design can't work in multi-project.
[~hit_lacus] PTAL
[Design
Doc|https://docs.google.com/document/u/1/d/1QfTn5dBcHeY2EAnMUS2V0sqj9dpFQwmVuAZon7wCtns/edit#]
> stream config lost when create table with same table_name in diff project.
> --------------------------------------------------------------------------
>
> Key: KYLIN-4921
> URL: https://issues.apache.org/jira/browse/KYLIN-4921
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
> Reporter: Kun Liu
> Assignee: Kun Liu
> Priority: Major
> Fix For: v3.1.2
>
>
> In project A, I have created a stream table test_table.
> When I create the table test_table in project B, the table can't be created,
> but the source config of table in project A lost.
>
> The reason:
> In the `saveStreamingConfig` function, if creating table failed, the config
> will be dropped in finally phase.
>
>
> By design:
> In order to identity one stream table `StreamingSourceConfig` in the
> metastore, kylin use the key
> `ResourceStore.STREAMING_V2_RESOURCE_ROOT + "/" + name + ".json"` as the
> identifier.
> But this design can't work in multi-project.
> [~hit_lacus] PTAL
>
> [Design
> Doc|https://docs.google.com/document/u/1/d/1QfTn5dBcHeY2EAnMUS2V0sqj9dpFQwmVuAZon7wCtns/edit#]
>
> Design:
> h1. The Rowkey of the Stream source config
>
> h1. Origin design
>
> In the project `project_test`, we create a real-time stream table with the
> name of `stream_table`, which will create two metadata, one the `tableDesc`
> and another is `streamSourceConfig`.
>
> The `tableDesc` is stored in the hbase by the path
> `table_prefix/tablename--projectname.json`, but the `streamSourceConfig` is
> stored in the hbase by the path `stream_source_prefix/tablename.json`. The
> action of creating tables with the same name in different projects is not
> allowed.
>
> h1. New design
>
> The new rowkey for storing the `streamSourceConfig` is
> `stream_source_prefix/tablename--projectname.json`.
>
> h1. How to deal with the compatibility in kylin
> h2. The type of RowKey
> * NewRowKey: `stream_source_prefix/tablename--projectname.json`
> * OldRowKey: `stream_source_prefix/tablename.json`
> h2. The type of operation
> * save source
> * update source
> * query/get source
> * delete
> h3. saveStreamingConfig:
> * store the stream source config with NewRowKey
> h3. removeStreamingConfig:
> * if the source config exist in the NewRowKey, delete the source config
> using the NewRowKey
> * if the source config exist in the OldRowKey, delete the source config
> using the OldRowKey
> h3. updateStreamingConfig:
> * removeStreamingConfig
> * saveStreamingConfig
>
> h3. reloadStreamingConfigLocal/queryStreamingConfig
> Hypothesis: The stream config must exist.
>
> * check NewRowKey, if the source config exists, return the object.
> * if the source config doesn't exist, and the source config exists in the
> OldRowKey
> * get the Source config and update the project name of the source config
> * delete the source config in the OldRowKey and resave the source config in
> the NewRowKey
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
