[
https://issues.apache.org/jira/browse/GOBBLIN-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Kumar updated GOBBLIN-1965:
------------------------------------
Description:
*Context :*
Need additional Hive data movement CDC check improvement to support table regex
lookup.
To support different whitelist pattern data movement.
Time based lookback support cutOffDate, referenceDate.
h3. *Hive CDC Asks*
||Description||Example value||
|To support different whitelist pattern data movement| # true -> Enable dynamic
table data movement with regex , datetime format .|
|hive databases, tables to be moved| # db1.table'yyyyMMdd' -> only db1.with
datatime format passed.
# db1.table1'yyyyMMdd',db2.table2'yyyyMMdd' -> combine expressions for
different databases with comma.
# db1.table1'yyyyMMdd'\|table2'yyyy-MM-dd' -> combine expressions for same
database with "\|".|
|Value can be +ve/-ve. Table older than this value will not be processed.| #
'3d' -> startDateTime < endDateTime; and the data movement happens in forward
order: oldest data first.
# '-3d' -> startDateTime > endDateTime; and the data movement happens in
reverse order: latest data first|
|Value can be +ve/-ve. Table older than this value will not be processed.| #
'20211213' -> lookback < 0 cutOffDate would be the startDateTime
# '20211213' -> lookback >=0 cutOffDate would be the endDateTime|
|Value can be +ve/-ve. Table older than this value will not be processed.| #
'20211213' -> lookback < 0 referenceDate would be the endDateTime
# '20211213' -> lookback >=0 referenceDate would be the startDateTime|
was:
*Context :*
Need additional Hive data movement CDC check improvement to support table regex
lookup.
> Need additional Hive data movement CDC check improvement to support table
> regex lookup
> --------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1965
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1965
> Project: Apache Gobblin
> Issue Type: Bug
> Components: misc
> Affects Versions: 0.15.0
> Reporter: Apekshit Kumar
> Priority: Minor
>
> *Context :*
> Need additional Hive data movement CDC check improvement to support table
> regex lookup.
> To support different whitelist pattern data movement.
> Time based lookback support cutOffDate, referenceDate.
> h3. *Hive CDC Asks*
> ||Description||Example value||
> |To support different whitelist pattern data movement| # true -> Enable
> dynamic table data movement with regex , datetime format .|
> |hive databases, tables to be moved| # db1.table'yyyyMMdd' -> only db1.with
> datatime format passed.
> # db1.table1'yyyyMMdd',db2.table2'yyyyMMdd' -> combine expressions for
> different databases with comma.
> # db1.table1'yyyyMMdd'\|table2'yyyy-MM-dd' -> combine expressions for same
> database with "\|".|
> |Value can be +ve/-ve. Table older than this value will not be processed.| #
> '3d' -> startDateTime < endDateTime; and the data movement happens in forward
> order: oldest data first.
> # '-3d' -> startDateTime > endDateTime; and the data movement happens in
> reverse order: latest data first|
> |Value can be +ve/-ve. Table older than this value will not be processed.| #
> '20211213' -> lookback < 0 cutOffDate would be the startDateTime
> # '20211213' -> lookback >=0 cutOffDate would be the endDateTime|
> |Value can be +ve/-ve. Table older than this value will not be processed.| #
> '20211213' -> lookback < 0 referenceDate would be the endDateTime
> # '20211213' -> lookback >=0 referenceDate would be the startDateTime|
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
