Re: RocksDB checkpointing dir per TM

There is also state.backend.rocksdb.localdir. Oddly, I can find the documentation for it in the 1.5 docs , but not in the 1.6 docs

Re: Custom Trigger + SQL Pattern

following up on the actual question - is there a way to register a keyedstream as table(s) and have a trigger per key? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink-1.6.1 :: HighAvailability :: ZooKeeperRunningJobsRegistry

Hi Till, thanks for your reply! here is the issue ticket: https://issues.apache.org/jira/browse/FLINK-10694 Kind Regards, Mike Pryakhin > On 26 Oct 2018, at 18:29, Till Rohrmann wrote: > > Hi Mike, > > thanks for reporting this issue. I

flink-1.6.1 :: job deployment :: detached mode

Hi community! Righ after I've upgraded flink up to flink-1.6.1 I get an exception during job deployment as a YARN cluster. The job is submitted with zookeper HA enabled, in detached mode. The flink yaml contains the following properties: high-availability: zookeeper

Re: Flink-1.6.1 :: HighAvailability :: ZooKeeperRunningJobsRegistry

Hi Mike, thanks for reporting this issue. I think you're right that Flink leaves some empty nodes in ZooKeeper. It seems that we don't delete the node with all its children in ZooKeeperHaServices#closeAndCleanupAllData. Could you please open a JIRA issue to in order to fix it? Thanks a lot!

Re: Flink-1.6.1 :: HighAvailability :: ZooKeeperRunningJobsRegistry

Hi Andrey, Thanks a lot for your reply! > What was the full job life cycle? 1. The job is deployed as a YARN cluster with the following properties set high-availability: zookeeper high-availability.zookeeper.quorum: high-availability.zookeeper.storageDir: hdfs:///

Re: 答复: Flink1.6.0 submit job and got "No content to map due to end-of-input" Error

Hi, I'm running Flink 1.5.4 and all dependencies in the job rely on 1.5.4. However, I still get this error. According to the JIRA issue it should be fixed in 1.5.4 as well. Since I'm using Apache Beam to build the jar, I can't move to version 1.6.x. What could it be? Cheers, Jeroen On

Re: Error with custom `SourceFunction` wrapping `InputFormatSourceFunction` (Flink 1.6)

Hey, Not sure how convo threading works on this list, so in case the folks CC'd missed my other response, here's some more info: First, I appreciate everyone's help! Thank you! I wrote several wrappers to try and debug this, including one which is an exact copy of `InputFormatSourceFunction`

Re: Flink Task Allocation on Nodes

Hi, There are couple of reasons: - Easier resource allocation and isolation: one faulty job doesn't affect another. - Mix and match of Flink version: you can leave the old stable jobs run with the old Flink version, and use the latest version of Flink for new jobs. - Faster metrics

HA jobmanagers redirect to ip address of leader instead of hostname

Hi, I'm having some troubles with Flink jobmanagers in a HA setup within OpenShift. I have three jobmanagers, a Zookeeper cluster and a loadbalancer (Openshift/Kubernetes Route) for the web ui / rest server on the jobmanagers. Everything works fine, as long as the loadbalancer connects to

Re: Accumulating a batch

Hi Hequn, Thank you for the info! Much appreciated! Do you know if a GroupBy Window could be used to buffer until an arbitrary record came through? Best, Austin On Thu, Oct 25, 2018, 9:17 PM Hequn Cheng wrote: > Hi Austin, > > You can use GroupBy Window[1], such as TUMBLE Window. The size of

RichInputFormat working differently in eclipse and in flink cluster

Hi all, I have implemented RichInputFormat for reading result of aggregation queries in Elasticsearch. There are around 10 buckets, which are of type json array. Note: This is one time response. My idea here is to iterate these arrays in parallel. Here is the pseudo code. public void

Re: RocksDB checkpointing dir per TM

Thanks! On Fri 26 Oct, 2018, 3:31 PM Andrey Zagrebin, wrote: > Hi Taher, > > TMs keep state locally while running, in this case RocksDB files already > belong to TM. > You can point it to the same NVME disk location on each node, relevant > Flink options here are: > - io.tmp.dirs > -

Re: Flink Task Allocation on Nodes

Hi all, In the mode of on yarn, a node may contain more than one container, is there a scheme for assigning tasks to different nodes. the version is 1.4.2 Thanks for your assistance. Sayat Satybaldiyev 于2018年10月26日周五 下午3:50写道： > Thanks for the advice, Klein. Could you please share more

Re: KafkaException or ExecutionStateChange failure on job startup

Hi Dominik Setting that bit of configuration seems to have done the trick for the MXBean exception. Many thanks for your help. Best regards, Mark On Tue, 23 Oct 2018 at 14:41, Dominik Wosiński wrote: > Hey Mark, > > Do You use more than 1 Kafka consumer for Your jobs? I think this relates >

Re: RocksDB checkpointing dir per TM

Hi Taher, TMs keep state locally while running, in this case RocksDB files already belong to TM. You can point it to the same NVME disk location on each node, relevant Flink options here are: - io.tmp.dirs - taskmanager.state.local.root-dirs This data is transient and has temporary nature. It

Re: Flink yarn -kill

Hi Mike, 'yarn -kill’ is out of Flink control. Currently, there is no hook provided by Flink for this. Maybe yarn has some feature for this kind of cleanup, then savepoint could be taken there. Best, Andrey > On 25 Oct 2018, at 15:44, Mikhail Pryakhin wrote: > > Hi Flink community, > Could

Re: Flink-1.6.1 :: HighAvailability :: ZooKeeperRunningJobsRegistry

Hi Mike, What was the full job life cycle? Did you start it with Flink 1.6.1 or canceled job running with 1.6.0? Was there a failover of Job Master while running before the cancelation? What version of Zookeeper do you use? Flink creates child nodes to create a lock for the job in Zookeeper.

Unbalanced Kafka consumer consumption

Hi, We are experience issues scaling our Flink application and we have observed that it may be because Kafka messages consumption is not balanced across partitions. The attached image (lag per partition) shows how only one partition consumes messages (the blue one in the back) and it wasn't until

RocksDB checkpointing dir per TM

Hi All, Our current cluster configuration uses one HDD which is mainly for root and an other NVME disk per node, [1]we want make sure all TMs write their own RocksDB files to the NVME disk only, how do we do that? [2] Is it also possible to specify multiple directories per TMs so that