[
https://issues.apache.org/jira/browse/HUDI-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-2271:
----------------------------
Description:
Follow-up items after HUDI-1138:
* Implement upgrade/downgrade, rollback regarding markers
** Rollback should be based on the markers in the file system, with marker
type indicated by MARKERS.type, or assumed to be "direct", if no MARKERS.type
** New write follows the `hoodie.write.markers.type` config to create markers
* Verify multiple concurrent writers with timeline-server-based markers
* Let's ensure the parallelism passed for local engine context is reasonable
* Fix `Task not serializable` due to new APIs in FSUtils for recursive, level
by level listing (`java.io.NotSerializableException:
org.apache.hudi.common.fs.FSUtils$$Lambda$4224/1845791682`)
* Optimize fs.exists(dirPath) call in marker creation in DirectMarkerFiles
* Optimize future timeouts due to I/O failure in the marker handler
* Test new marker strategy for all engines (spark, flink and java) w/o any
additional changes
was:
Follow-up items after HUDI-1138:
* Optimize fs.exists(dirPath) call in marker creation in DirectMarkerFiles
* Test new marker strategy for all engines (spark, flink and java) w/o any
additional changes
> Follow-up items for timeline-server-based marker files
> ------------------------------------------------------
>
> Key: HUDI-2271
> URL: https://issues.apache.org/jira/browse/HUDI-2271
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Priority: Major
>
> Follow-up items after HUDI-1138:
> * Implement upgrade/downgrade, rollback regarding markers
> ** Rollback should be based on the markers in the file system, with marker
> type indicated by MARKERS.type, or assumed to be "direct", if no MARKERS.type
> ** New write follows the `hoodie.write.markers.type` config to create markers
> * Verify multiple concurrent writers with timeline-server-based markers
> * Let's ensure the parallelism passed for local engine context is reasonable
> * Fix `Task not serializable` due to new APIs in FSUtils for recursive,
> level by level listing (`java.io.NotSerializableException:
> org.apache.hudi.common.fs.FSUtils$$Lambda$4224/1845791682`)
> * Optimize fs.exists(dirPath) call in marker creation in DirectMarkerFiles
> * Optimize future timeouts due to I/O failure in the marker handler
> * Test new marker strategy for all engines (spark, flink and java) w/o any
> additional changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
