Re: Savepoint Location from Flink REST API

2020-04-01 Thread Aaron Langford
2. Failure > > { > "status": { > "id": "completed" > }, > "savepoint": { > "failure-cause": { > "class": "string", > "stack-trace": "string", > &

Re: How to enforce ACLs on Flink JobManager/ApplicationMaster URL on Yarn

2020-03-30 Thread Aaron Langford
I'd be curious to see how others have done this, but our setup restricts network access to machines in the YARN cluster to a jump box. Access to Flink job manager is limited to whoever can ssh to that box, and that is controlled with an Ansible playbook. Additionally, we have a list of users

Re: Savepoint Location from Flink REST API

2020-03-20 Thread Aaron Langford
index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html > > Meanwhile, I've created an issue to update the docs: > https://issues.apache.org/jira/browse/FLINK-16696 > > Regards, > Roman > > > On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford > w

Savepoint Location from Flink REST API

2020-03-19 Thread Aaron Langford
Hey Flink Community, I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be

Re: Location of flink-s3-fs-hadoop plugin (Flink 1.9.0 on EMR)

2020-01-24 Thread Aaron Langford
s > - Cannot find hdfs-site configuration-file path in > Flink config. > > > > > > *From: *Aaron Langford > *Date: *Thursday, January 23, 2020 at 12:22 PM > *To: *Senthil Kumar > *Cc: *Yang Wang , "user@flink.apache.org" < > user@fl

Re: Location of flink-s3-fs-hadoop plugin (Flink 1.9.0 on EMR)

2020-01-23 Thread Aaron Langford
fication": "flink-log4j-yarn-session", > "Properties": { > "log4j.rootLogger": "DEBUG,stdout" > } > }] > }] > }' On Thu, Jan 23, 2020 at 11:03 AM Senthil Kumar wrote: > Could

Re: Replacing a server in Zookeeper Quorum

2020-01-21 Thread Aaron Langford
tion. One quick solution is > to use a static ip for EMR > master node[1]. > > > Best, > Yang > > > [1]. > https://aws.amazon.com/premiumsupport/knowledge-center/static-private-ip-master-node-emr/?nc1=h_ls > > Aaron Langford 于2020年1月22日周三 上午1:48写道: > >> Hello F

Re: Location of flink-s3-fs-hadoop plugin (Flink 1.9.0 on EMR)

2020-01-21 Thread Aaron Langford
Senthil, One of the key steps in debugging this for me was enabling debug level logs on my cluster, and then looking at the logs in the resource manager. The failure you are after happens before the exceptions you have reported here. When your Flink application is starting, it will attempt to

Replacing a server in Zookeeper Quorum

2020-01-21 Thread Aaron Langford
Hello Flink Community, I'm working on a HA setup of Flink 1.8.1 on AWS EMR and have some questions about how Flink interacts with Zookeeper when one of the servers in the quorum specified in flink-conf.yaml goes down and is replaced by a machine with a new IP address. Currently, I configure

Rich Function Thread Safety

2019-12-18 Thread Aaron Langford
Hello Flink Community, I'm hoping to verify some understanding: If I have a function with managed state, I'm wondering if a checkpoint will ever be taken while a function is mutating state. I'll try to illustrate the situation I'm hoping to be safe from: Happy Path: t0 -> processFunction

Re: MapState with List Type for values

2019-12-18 Thread Aaron Langford
to search the map to know we already queued two > . In this time we could produce {, 咽} and set previous map state as < > , 1> . If you could follow this logic, the previous > serialize/deserialize of Seq could be greatly reduced. > > Best > Yun Tang > --

MapState with List Type for values

2019-12-17 Thread Aaron Langford
Hello Flink Community, I have a question about using MapState with lists as values. Here is a description of the use case: I have an operator over a keyed stream where for each record that comes, it needs to look into some state to determine if a value has arrived or not. If the value has not

Per Operator State Monitoring

2019-11-22 Thread Aaron Langford
Hey Flink Community, I'm working on a Flink application where we are implementing operators that extend the RichFlatMap and RichCoFlatMap interfaces. As a result, we are working directly with Flink's state API (ValueState, ListState, MapState). Something that appears to be extremely valuable is