Re: Flink keyed stream windows

Clarification: Its 30 Seconds not 30 minutes. On Mon, Aug 13, 2018 at 3:20 PM Garvit Sharma wrote: > Hi, > > I am working on a use case where I have a stream of users active locations > and I want to store(by hitting an HTTP API) the latest active location for > each of the uni

Flink keyed stream windows

stream to solve this problem. Please help! Thanks, -- Garvit Sharma github.com/garvitlnmiit/ No Body is a Scholar by birth, its only hard work and strong determination that makes him master.

Re: Cleaning of state snapshot in state backend(HDFS)

y anymore unless you add it > again. > > Best, Sihua > > > On 06/21/2018 16:04，Garvit Sharma > wrote： > > Now, after clearing state for a key, I don't want that redundant data in > the state backend. This is my concern. > > Please let me know if there are any gaps.

Re: Cleaning of state snapshot in state backend(HDFS)

Now, after clearing state for a key, I don't want that redundant data in the state backend. This is my concern. Please let me know if there are any gaps. Thanks, On Thu, Jun 21, 2018 at 1:31 PM Garvit Sharma wrote: > I am maintaining state data for a key in ValueState. As per [0] I

Re: Cleaning of state snapshot in state backend(HDFS)

r a key-value pair > of the state currently, you can't cleared the whole state currently. > > Best, Sihua > > On 06/21/2018 15:41，Garvit Sharma > wrote： > > So, would it delete all the files in HDFS associated with the cleared > state? > > On Thu, Jun 21, 2018 at 12:58 PM si

Re: Cleaning of state snapshot in state backend(HDFS)

eleted but never be changed. Maybe Stefan(cc) could give you > more professional information and plz correct me if I'm incorrect. > > Best, Sihua > On 06/21/2018 14:40，Garvit Sharma > wrote： > > Hi, > > Consider a managed keyed state backed by HDFS with checkpointin

Cleaning of state snapshot in state backend(HDFS)

on clearing the keyed state? -- Garvit Sharma github.com/garvitlnmiit/ No Body is a Scholar by birth, its only hard work and strong determination that makes him master.

Re: Exception while submitting jobs through Yarn

wrote: > Since you're using a vendor's distro, I would suggest asking on their user > forum. > > Cheers > > Original message ---- > From: Garvit Sharma > Date: 6/19/18 3:34 AM (GMT-08:00) > To: trohrm...@apache.org > Cc: Amit Jain , Chesnay Schepler , &g

Re: Exception while submitting jobs through Yarn

Any help on this? On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma wrote: > Yes, it is. > > On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann > wrote: > >> Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to ` >> /usr/hdp/2.6.3.0-235/hadoop/client/xercesIm

Re: Exception while submitting jobs through Yarn

Yes, it is. On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann wrote: > Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to ` > /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`? > > On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma wrote: > >> I don't th

Re: Exception while submitting jobs through Yarn

I don't think I can access core-default as it comes with Hadoop jar On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann wrote: > Hmm, could you check whether core-default.xml contains any suspicious > entries? Apparently xerces:2.9.1 cannot read it. > > On Mon, Jun 18, 2018 at 3:40 PM G

Re: Exception while submitting jobs through Yarn

f4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-

Re: Exception while submitting jobs through Yarn

error for core-default.xml is interesting. >> >> Flink doesn't have this file. Probably it came with Yarn. Please check >> the hadoop version Flink was built with versus the hadoop version in your >> cluster. >> >> Thanks >> >> Original m

Re: Exception while submitting jobs through Yarn

ient logs with us. > > Cheers, > Till > > On Fri, Jun 15, 2018 at 4:54 PM Garvit Sharma wrote: > >> Yes, I did. >> >> On Fri, Jun 15, 2018 at 6:17 PM Till Rohrmann >> wrote: >> >>> Hi Garvit, >>> >>> have you exported t

Re: Exception while submitting jobs through Yarn

p-classpath-discovery > > Cheers, > Till > > On Fri, Jun 15, 2018 at 2:22 PM Garvit Sharma wrote: > >> Does someone has any idea how to get rid if the above parse exception >> while submitting flink job to Yarn. >> >> Already searched on the internet, could not f

Re: Exception while submitting jobs through Yarn

Does someone has any idea how to get rid if the above parse exception while submitting flink job to Yarn. Already searched on the internet, could not find any solution to it. Please help. On Fri, Jun 15, 2018 at 9:15 AM Garvit Sharma wrote: > Thanks Chesnay, Now it is connect

Re: Exception while submitting jobs through Yarn

ed jar. > For a simple test, put your jar into the /lib folder before submitting it. > > On 14.06.2018 06:56, Garvit Sharma wrote: > > Can someone please tell why am I facing this? > > On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma > wrote: > >> Hi, >> >> I am us

Re: Exception while submitting jobs through Yarn

Can someone please tell why am I facing this? On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma wrote: > Hi, > > I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs through > Yarn, but I am getting the below exception : > > java.lang.NoClassDefFoundError: > com

Re: Any remote opportunity to work on Flink project?

yone is looking for freelancers around any of > their Flink projects. > > -- > Thanks > Deepak > -- Garvit Sharma github.com/garvitlnmiit/ No Body is a Scholar by birth, its only hard work and strong determination that makes him master.

Re: Implementing a “join” between a DataStream and a “set of rules”

way to do this kind of a join? > > 2.If not, is it safe to add additional parallelism by creating > new threads inside each flatMap operation, on top of what Flink is already > doing? > > > > Thanks in advance! > > Turar > > > -- Garvit Sharma github.com/garvitlnmiit/ No Body is a Scholar by birth, its only hard work and strong determination that makes him master.

Re: Checkpointing on cluster shutdown

to manually trigger savepoints in these cases? >> Also am I correct in my understanding that if a taskmanager goes down >> first, there is no way the TaskManager can trigger the checkpoint on its >> own? >> >> >> > -- Garvit Sharma github.com/garvitlnmiit/ No Body is a Scholar by birth, its only hard work and strong determination that makes him master.

Re: ConcurrentModificationException while accessing managed keyed state

I am not creating a custom thread to operate the state. Code snippet : https://gist.github.com/garvitlnmiit/10db9d4b6eb41135332fba13d908e36c I will reply back with the Stacktrace soon. Thanks, On Sat, Jun 2, 2018 at 6:49 PM, aitozi wrote: > Hi Garvit Sharma, > > Flink run with per

ConcurrentModificationException while accessing managed keyed state

help me in handling the case according to my use case. Thanks, -- Garvit Sharma github.com/garvitlnmiit/ No Body is a Scholar by birth, its only hard work and strong determination that makes him master.

Re: Regarding Keyed State of Connected Stream

e thread > sync. > > Best, Sihua > > > > > > > 在2018年05月25日 10:26，Garvit Sharma <garvit...@gmail.com> 写道： > Hi, > > Let's consider, I have two keyed streams one for rules and another for > data and I have created a connected stream. > > I am mai

Regarding Keyed State of Connected Stream

while updating the state or not? I need to understand. Please let me know. Thanks, -- Garvit Sharma github.com/garvitlnmiit/ No Body is a Scholar by birth, its only hard work and strong determination that makes him master.