You can definitely use absolute URIs to access two clusters. The configuration just has to be the union of multiple HDFS clusters (e.g., the NameNode lists)
Accessing both secure and non-secure clusters are fairly tricky but it can be done. AFAIK Isolating the Hadoop configuration will require a lot of changes in Flink itself today. ~Haohui On Thu, Aug 24, 2017 at 5:30 AM, Vijay Srinivasaraghavan <vijikar...@yahoo.com.invalid> wrote: > I think it may not work in scenario where Hadoop security is enabled and each > HCFS setup is configured differently, unless if there is a way to isolate the > Hadoop configurations used in this case? > > Regards, > Vijay > > Sent from my iPhone > >> On Aug 24, 2017, at 2:51 AM, Stephan Ewen <se...@apache.org> wrote: >> >> Hi! >> >> I think it can work if you fully qualify the URIs. >> >> For the checkpoint configuration, specify one namenode (in the >> flink-config.yml or in the constructor of the state backend). >> Example: statebackend.fs.checkpoint.dir: >> hdfs://dfsOneNamenode:port/flink/checkpoints >> >> For the result (for example rolling sink), configure it with >> hdfs://dfsTwoNamenode:otherport/flink/result >> >> Is that what you are looking for? >> >> Stephan >> >> >> On Thu, Aug 24, 2017 at 11:47 AM, Stefan Richter < >> s.rich...@data-artisans.com> wrote: >> >>> Hi, >>> >>> I don’t think that this is currently supported. If you see a use case for >>> this (over creating different root directories for checkpoint data and >>> result data) then I suggest that you open a JIRA issue with a new feature >>> request. >>> >>> Best, >>> Stefan >>> >>>> Am 23.08.2017 um 20:17 schrieb Vijay Srinivasaraghavan < >>> vijikar...@yahoo.com>: >>>> >>>> Hi Ted, >>>> >>>> I believe HDFS-6584 is more of an HDFS feature supporting archive use >>> case through some policy configurations. >>>> >>>> My ask is that I have two distinct HCFS File systems which are >>> independent but the Flink job will decide which one to use for sink while >>> the Flink infrastructure is by default configured with one of these HCFS as >>> state backend store. >>>> >>>> Hope this helps. >>>> >>>> Regards >>>> Vijay >>>> >>>> >>>> On Wednesday, August 23, 2017 11:06 AM, Ted Yu <yuzhih...@gmail.com> >>> wrote: >>>> >>>> >>>> Would HDFS-6584 help with your use case ? >>>> >>>> On Wed, Aug 23, 2017 at 11:00 AM, Vijay Srinivasaraghavan < >>>> vijikar...@yahoo.com.invalid <mailto:vijikar...@yahoo.com.invalid>> >>> wrote: >>>> >>>>> Hello, >>>>> Is it possible for a Flink cluster to use multiple HDFS repository >>> (HDFS-1 >>>>> for managing Flink state backend, HDFS-2 for syncing results from user >>>>> job)? >>>>> The scenario can be viewed in the context of running some jobs that are >>>>> meant to push the results to an archive repository (cold storage). >>>>> Since the hadoop configuration is static, I am thinking it is hard to >>>>> achieve this but I could be wrong. >>>>> Please share any thoughts. >>>>> RegardsVijay >>>> >>>> >>> >>> >
