Re: Need help with finding inner workings of watermark stream idleness
Thanks, Seth. Yea this looks perfect. I had a feeling I'd need to get deep into things, and no time like the present haha. May ask for more guidance with those inner workings to get a bit of a road map. But that gets into the feature idea and beyond the scope of this thread's original question so I'll just do that in a jira ticket in a bit. Just wanted this so I could structure the ticket and plan of attack better. Thanks!! On Tue, Feb 1, 2022, 2:03 PM Seth Wiesman wrote: > Hi Jeff, > > I think the class you're looking for is StatusWatermarkValve. Note that > this is fairly deep into the runtime stack. > > Seth > > On Tue, Feb 1, 2022 at 2:34 PM Jeff Carter wrote: > > > Thanks, Till. > > > > That definitely helps a bit. I'm still not seeing where there is some > idle > > variable that the output.markIdle is setting to true (or whatever it > sets). > > Like the ideal thing would be if there is just some "output.isIdle()" > that > > could be called to know if the stream is or isnt idle. Since that doesn't > > exist, what is the variable in "output" that dictates if it is idle or > not > > that that I'd just have to make an isIdle() method to make its state > > visible to other code. > > > > I see the checkIfIdle() method in the code (in at least the testing > piece) > > you pointed out, but that seems like it's just a way to set a timer and > > check if the idle state should be set or not. But I dont know if that's > > setting some isIdle variable or if it's just checked and calculated > > everytime and that method is basically the variable I'm looking for. But > > that might just be my confusion. > > > > On Tue, Jan 11, 2022, 11:05 AM Till Rohrmann > wrote: > > > > > Hi Jeff, > > > > > > I think this happens in the WatermarksWithIdleness [1]. > > > > > > [1] > > > > > > > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarksWithIdleness.java#L73 > > > > > > Cheers, > > > Till > > > > > > On Tue, Jan 11, 2022 at 6:05 PM Jeff Carter > > wrote: > > > > > > > I'm looking into making a feature for flink related to watermarks and > > am > > > > digging into the inner watermark mechanisms, specifically with > > idleness. > > > > I'm familiar with idleness, but digging into the root code I can only > > get > > > > to where idlenessTimeout gets set in > > WatermarkStrategyWithIdleness.java. > > > > > > > > But what I'm looking for the pieces beyond that. If I set the > idleness > > > to > > > > 500 milliseconds, where in the code does it actually go "I haven't > > seen a > > > > message in 500 milliseconds. I'm setting this stream to idle."? > > > > > > > > The reason being that what I'm thinking of would need to be able to > see > > > if > > > > any streams are marked idle, and if so react accordingly. > > > > > > > > Thanks for any help in advance. > > > > > > > > > >
Re: Need help with finding inner workings of watermark stream idleness
Thanks, Till. That definitely helps a bit. I'm still not seeing where there is some idle variable that the output.markIdle is setting to true (or whatever it sets). Like the ideal thing would be if there is just some "output.isIdle()" that could be called to know if the stream is or isnt idle. Since that doesn't exist, what is the variable in "output" that dictates if it is idle or not that that I'd just have to make an isIdle() method to make its state visible to other code. I see the checkIfIdle() method in the code (in at least the testing piece) you pointed out, but that seems like it's just a way to set a timer and check if the idle state should be set or not. But I dont know if that's setting some isIdle variable or if it's just checked and calculated everytime and that method is basically the variable I'm looking for. But that might just be my confusion. On Tue, Jan 11, 2022, 11:05 AM Till Rohrmann wrote: > Hi Jeff, > > I think this happens in the WatermarksWithIdleness [1]. > > [1] > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarksWithIdleness.java#L73 > > Cheers, > Till > > On Tue, Jan 11, 2022 at 6:05 PM Jeff Carter wrote: > > > I'm looking into making a feature for flink related to watermarks and am > > digging into the inner watermark mechanisms, specifically with idleness. > > I'm familiar with idleness, but digging into the root code I can only get > > to where idlenessTimeout gets set in WatermarkStrategyWithIdleness.java. > > > > But what I'm looking for the pieces beyond that. If I set the idleness > to > > 500 milliseconds, where in the code does it actually go "I haven't seen a > > message in 500 milliseconds. I'm setting this stream to idle."? > > > > The reason being that what I'm thinking of would need to be able to see > if > > any streams are marked idle, and if so react accordingly. > > > > Thanks for any help in advance. > > >
Need help with finding inner workings of watermark stream idleness
I'm looking into making a feature for flink related to watermarks and am digging into the inner watermark mechanisms, specifically with idleness. I'm familiar with idleness, but digging into the root code I can only get to where idlenessTimeout gets set in WatermarkStrategyWithIdleness.java. But what I'm looking for the pieces beyond that. If I set the idleness to 500 milliseconds, where in the code does it actually go "I haven't seen a message in 500 milliseconds. I'm setting this stream to idle."? The reason being that what I'm thinking of would need to be able to see if any streams are marked idle, and if so react accordingly. Thanks for any help in advance.
Re: [DISCUSS] Dropping flink-storm?
+1 to drop it. On Fri, Sep 28, 2018, 7:25 PM Hequn Cheng wrote: > Hi, > > +1 to drop it. It seems that few people use it. > > Best, Hequn > > On Fri, Sep 28, 2018 at 10:22 PM Chesnay Schepler > wrote: > > > I'm very much in favor of dropping it. > > > > Flink has been continually growing in terms of features, and IMO we've > > reached the point where we should cull some of the more obscure ones. > > flink-storm, while interesting from a theoretical standpoint, offers too > > little value. > > > > Note that the bolt/spout wrapper parts of the part are still compatible, > > it's only topologies that aren't working. > > > > IMO compatibility layers only add value if they ease the migration to > > Flink APIs. > > * bolt/spout wrappers do this, but they will continue to work even if we > > drop it > > * topologies don't do this, so I'm not interested in then. > > > > On 28.09.2018 15:22, Till Rohrmann wrote: > > > Hi everyone, > > > > > > I would like to discuss how to proceed with Flink's storm > > > compatibility layer flink-strom. > > > > > > While working on removing Flink's legacy mode, I noticed that some > > > parts of flink-storm rely on the legacy Flink client. In fact, at the > > > moment flink-storm does not work together with Flink's new distributed > > > architecture. > > > > > > I'm also wondering how many people are actually using Flink's Storm > > > compatibility layer and whether it would be worth porting it. > > > > > > I see two options how to proceed: > > > > > > 1) Commit to maintain flink-storm and port it to Flink's new > architecture > > > 2) Drop flink-storm > > > > > > I doubt that we can contribute it to Apache Bahir [1], because once we > > > remove the legacy mode, this module will no longer work with all newer > > > Flink versions. > > > > > > Therefore, I would like to hear your opinion on this and in particular > > > if you are using or planning to use flink-storm in the future. > > > > > > [1] https://github.com/apache/bahir-flink > > > > > > Cheers, > > > Till > > > > > > >
Re: Am I forgetting to do something to get pull request accepted?
Ok, thanks! On Thu, Jun 21, 2018, 1:18 AM Chesnay Schepler wrote: > The Flink project currently receives too many pull requests / has to few > reviewers to ensure timely reviews. > > I assume you are talking about https://github.com/apache/flink/pull/5876 > If this is correct, something you could already do is squash all commits > and rebase against master to get rid of the merge commits. > The new InputFormat also shares a lot of code with the existing > CassandraOutputFormat. Shared portions could be moved in abstract > CassandraInputFormatBase class. > > On 20.06.2018 20:50, jpcarter...@gmail.com wrote: > > I just want to make sure I am not missing anything to get my pull > request accepted. All tests are passing so I figured it was going to be a > short time after that. Is there anything else that I need to do or a reason > like there is a freeze on accepting pull requests for the moment? Here is > the link to it in Travis IO: > https://travis-ci.org/Jicaar/flink/builds/386283019 > > > >
