[
https://issues.apache.org/jira/browse/HUDI-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2860:
--------------------------------------
Description:
Make timeline server work with multiple concurrent writers.
As of now, if an executor is lagging wrt timeline server (timeline server
refreshes its state for every call if timeline has moved), we throw an
exception and executor falls back to secondary which will list the file system.
Related ticket: https://issues.apache.org/jira/browse/HUDI-2761
We want to revisit this code and see how can we make timeline server work with
multi-writer scenario.
Few points to consider:
1. Executors should try to call getLatestBaseFilesOnOrBefore() instead of
getLatestBaseFiles(). Not calls has to be fixed. the ones doing conflict
resolutions, might have to get the latest snapshot always.
2. Fix async services to use separate write client in deltastreamer flow
3. Revist every call from executor and set "REFRESH" param on only when matters.
4. Sharing embedded timeline server.
5. Check for any holes. when C100 and C101 concurrently started and C101
finishes early, if C100 makes getLatestBaseFileOnOrBefore(), do we return base
files from C101?
was:
Make timeline server work with multiple concurrent writers.
As of now, if an executor is lagging wrt timeline server (timeline server
refreshes its state for every call if timeline has moved), we throw an
exception and executor falls back to secondary which will list the file system.
We want to revisit this code and see how can we make timeline server work with
multi-writer scenario.
Few points to consider:
1. Executors should try to call getLatestBaseFilesOnOrBefore() instead of
getLatestBaseFiles(). Not calls has to be fixed. the ones doing conflict
resolutions, might have to get the latest snapshot always.
2. Fix async services to use separate write client in deltastreamer flow
3. Revist every call from executor and set "REFRESH" param on only when matters.
4. Sharing embedded timeline server.
5. Check for any holes. when C100 and C101 concurrently started and C101
finishes early, if C100 makes getLatestBaseFileOnOrBefore(), do we return base
files from C101?
> Make timeline server work with multiple concurrent writers
> ----------------------------------------------------------
>
> Key: HUDI-2860
> URL: https://issues.apache.org/jira/browse/HUDI-2860
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
> Reporter: sivabalan narayanan
> Priority: Blocker
> Fix For: 0.11.0
>
>
> Make timeline server work with multiple concurrent writers.
> As of now, if an executor is lagging wrt timeline server (timeline server
> refreshes its state for every call if timeline has moved), we throw an
> exception and executor falls back to secondary which will list the file
> system.
>
> Related ticket: https://issues.apache.org/jira/browse/HUDI-2761
>
> We want to revisit this code and see how can we make timeline server work
> with multi-writer scenario.
>
> Few points to consider:
> 1. Executors should try to call getLatestBaseFilesOnOrBefore() instead of
> getLatestBaseFiles(). Not calls has to be fixed. the ones doing conflict
> resolutions, might have to get the latest snapshot always.
> 2. Fix async services to use separate write client in deltastreamer flow
> 3. Revist every call from executor and set "REFRESH" param on only when
> matters.
> 4. Sharing embedded timeline server.
> 5. Check for any holes. when C100 and C101 concurrently started and C101
> finishes early, if C100 makes getLatestBaseFileOnOrBefore(), do we return
> base files from C101?
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
