[
https://issues.apache.org/jira/browse/FLINK-33508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796058#comment-17796058
]
Anika Kelhanka commented on FLINK-33508:
----------------------------------------
Approach to update Flink's History Server logic to enable getting logs and data
from multiple directories at a time by using a path with wildcards (i.e glob
pattern) from HadoopFileSystem locations:
1. Flink's {{HistoryServerArchiveFetcher}} class currently uses the
HadoopFileSystem's {{listStatus}} API method which not resolve
patterns/wildcards in the history server file path.
2. Introduce a new method {{globStatus(Path pathPattern)}} in {{Flink's
FileSystem}} API.
3. Implement new Method in Flink's HadoopFileSystem class such that it
internally calls the [Hadoop's globStatus
func|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2217].
4. Finally, point {{HistoryServerArchiveFetcher}} to the new globStatus() API
instead of listStatus() for HadoopFileSystem.
> Support for wildcard paths in Flink History Server for multi cluster
> environment
> --------------------------------------------------------------------------------
>
> Key: FLINK-33508
> URL: https://issues.apache.org/jira/browse/FLINK-33508
> Project: Flink
> Issue Type: Improvement
> Reporter: Jayadeep Jayaraman
> Assignee: Jayadeep Jayaraman
> Priority: Major
> Labels: pull-request-available
>
> In Cloud users typically create multiple clusters which are ephemeral and
> want a single history server to look at historical jobs.
> To implement this history server needs to support wildcard paths and this
> change is to support such wildcard paths
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
