Re: [EXTERNAL] Re: Looking for help in configuring Swift as State Backend
I think you also need to specify a path for the checkpoint directory. Try to set state.checkpoints.dir: swift://spout-checkpoints.magellan/flink/checkpoints Cheers, Till On Wed, Apr 24, 2019 at 2:58 PM PoolakkalMukkath, Shakir < shakir_poolakkalmukk...@comcast.com> wrote: > Hi Till, Thanks for the response. Yes, I looks at the document. But still > trying to figure out > > > > Let me summaries my config and what I did > > > >1. Copied flink-swift-fs-hadoop-1.6.2.jar to lib >2. *flink-conf.yaml* > > > > > #== > > # Fault tolerance and checkpointing > > > #== > > > > # The backend that will be used to store operator state checkpoints if > > # checkpointing is enabled. > > # > > # Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the > > # . > > # > > state.backend: filesystem > > > > # Directory for checkpoints filesystem, when using any of the default > bundled > > # state backends. > > state.checkpoints.dir: swift://spout-checkpoints.magellan > > > > > #== > > # Hadoop > > > #== > > fs.hdfs.hadoopconf: /app/stream/flink-standalone/hadoop/ > > OR > > > > export HADOOP_CONF_DIR=/app/stream/flink-standalone/hadoop/ > > > > > > > >1. And have the core-site.xml in >HADOOP_CONF_DIR=/app/stream/flink-standalone/hadoop/ > > > > > > > > > > > > > > > > fs.swift.impl > > org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem > > > > > > > > > fs.swift.service.magellan.auth.url > > https://osvip-as-c01.ece.***.net:5000/v3 > > > > > > > > fs.swift.service.magellan.username > > *** > > > > > > > > fs.swift.service.magellan.password > > *** > > > > > > > > fs.swift.service.magellan.public > > true > > > > > > > > > > > > When I submit a job with Checkpointing enabled, getting the below error, > > > > java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > >at > org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36) > >at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) > >at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) > >at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) > >at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > >at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > >at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > >at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > Caused by: org.apache.flink.runtime.client.JobExecutionException: Could > not set up JobManager > >at > org.apache.flink.runtime.jobmaster.JobManagerRunner.(JobManagerRunner.java:176) > >at > org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:1058) > >at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:308) > >at > org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34) > >... 7 more > > Caused by: org.apache.flink.runtime.client.JobExecutionException: Could > not instantiate configured state backend > >at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:308) > >at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100) > >at > org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1151) > >at > org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1131) > >at > org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:294) > >at > org.apache.flink.runtime.jobmaster.JobManagerRunner.(JobManagerRunner.java:157) > >... 10 more > > Caused by: org.apache.flink.configuration.IllegalConfigurationException: > Invalid configuration for the state backend > >at > org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.createFromConfig(FsStateBackendFactory.java:48) > >at > org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:121) > >at > org.apache.flink.runtime.state.StateBackendLoader.fromApplicationOrConfigOrDefault(StateBackendLoader.java:222) > >at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:304) > >
Re: [EXTERNAL] Re: Looking for help in configuring Swift as State Backend
Hi Till, Thanks for the response. Yes, I looks at the document. But still trying to figure out Let me summaries my config and what I did 1. Copied flink-swift-fs-hadoop-1.6.2.jar to lib 2. flink-conf.yaml #== # Fault tolerance and checkpointing #== # The backend that will be used to store operator state checkpoints if # checkpointing is enabled. # # Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the # . # state.backend: filesystem # Directory for checkpoints filesystem, when using any of the default bundled # state backends. state.checkpoints.dir: swift://spout-checkpoints.magellan #== # Hadoop #== fs.hdfs.hadoopconf: /app/stream/flink-standalone/hadoop/ OR export HADOOP_CONF_DIR=/app/stream/flink-standalone/hadoop/ 1. And have the core-site.xml in HADOOP_CONF_DIR=/app/stream/flink-standalone/hadoop/ fs.swift.impl org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem fs.swift.service.magellan.auth.url https://osvip-as-c01.ece.***.net:5000/v3 fs.swift.service.magellan.username *** fs.swift.service.magellan.password *** fs.swift.service.magellan.public true When I submit a job with Checkpointing enabled, getting the below error, java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager at org.apache.flink.runtime.jobmaster.JobManagerRunner.(JobManagerRunner.java:176) at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:1058) at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:308) at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34) ... 7 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not instantiate configured state backend at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:308) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100) at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1151) at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1131) at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:294) at org.apache.flink.runtime.jobmaster.JobManagerRunner.(JobManagerRunner.java:157) ... 10 more Caused by: org.apache.flink.configuration.IllegalConfigurationException: Invalid configuration for the state backend at org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.createFromConfig(FsStateBackendFactory.java:48) at org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:121) at org.apache.flink.runtime.state.StateBackendLoader.fromApplicationOrConfigOrDefault(StateBackendLoader.java:222) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:304) ... 15 more Caused by: java.lang.IllegalArgumentException: Cannot use the root directory for checkpoints. at org.apache.flink.runtime.state.filesystem.AbstractFileStateBackend.validatePath(AbstractFileStateBackend.java:195) at org.apache.flink.runtime.state.filesystem.AbstractFileStateBackend.(AbstractFileStateBackend.java:109) at org.apache.flink.runtime.state.filesystem.AbstractFileStateBackend.(AbstractFileStateBackend.java:95) at org.apache.flink.runtime.state.filesystem.FsStateBackend.(FsStateBackend.java:319) at org.apache.flink.runtime.state.filesystem.FsStateBackend.(FsStateBackend.java:200) at
Re: Looking for help in configuring Swift as State Backend
Hi Shakir, have you checked out Flink's documentation for Filesystems [1]? What is the problem you are observing? [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html Cheers, Till On Tue, Apr 23, 2019 at 9:30 PM PoolakkalMukkath, Shakir < shakir_poolakkalmukk...@comcast.com> wrote: > Hi, > > > > I am looking for some help in configuring the Swift Fs as State Backend. I > am unable to configure it, let me know if anyone has prior done this or > knowledge to help me > > Do we still need to run an HDFS to use this feature ? > > > > Thanks, > > Shakir >
Looking for help in configuring Swift as State Backend
Hi, I am looking for some help in configuring the Swift Fs as State Backend. I am unable to configure it, let me know if anyone has prior done this or knowledge to help me Do we still need to run an HDFS to use this feature ? Thanks, Shakir
