[jira] [Created] (FLINK-5647) Fix RocksDB Backend Cleanup

2017-01-25 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5647: --- Summary: Fix RocksDB Backend Cleanup Key: FLINK-5647 URL: https://issues.apache.org/jira/browse/FLINK-5647 Project: Flink Issue Type: Bug

Re: Unregistering Managed State in Operator Backend

2017-01-24 Thread Aljoscha Krettek
Just a bit of clarification, the OperatorState stuff is independent of keyed state backends, i.e. even if you use RocksDB the operator state will not be stored in RocksDB, only keyed state is stored there. Right now, when an operator state (ListState) is empty we will still write some meta data ab

Re: Need help on understanding flink runtime and window function

2017-01-24 Thread Aljoscha Krettek
Hi,depending on which version of Flink you're using the answer changes. If you use Flink 1.1 AggregatingProcessingTimeWindowOperator should be responsible for executing that. In Flink 1.2 it should be WindowOperator. For a quick overview of how scheduling works in Flink you could look at this: htt

Re: [DISCUSS] (Not) tagging reviewers

2017-01-24 Thread Aljoscha Krettek
It seems I'm in a bit of a minority here but I like the @R tags. There are simply to many pull request for someone to keep track of all of them and if someone things that a certain person would be good for reviewing a change then tagging them helps them notice the PR. I think the tag should not me

[jira] [Created] (FLINK-5616) YarnPreConfiguredMasterHaServicesTest fails sometimes

2017-01-23 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5616: --- Summary: YarnPreConfiguredMasterHaServicesTest fails sometimes Key: FLINK-5616 URL: https://issues.apache.org/jira/browse/FLINK-5616 Project: Flink

Re: Slow duplicated tests

2017-01-10 Thread Aljoscha Krettek
again. > */ > > but he incorrect, because code of both test fully equal, > one difference line very similar on bug after refactoring with inserting > recordReuse > > testSpillingHashJoinWithMassiveCollisions > 353 while ((record = buildSide.next(record)) != null) { >

Re: [ANNOUNCE] Flink 1.1.4 Released

2016-12-25 Thread Aljoscha Krettek
Very nice. Good work, team! šŸ˜€ On Sat, Dec 24, 2016, 00:07 Fabian Hueske wrote: > Thank you Ufuk for your work as release manager and everybody who > contributed! > > Cheers, Fabian > > 2016-12-23 16:40 GMT+01:00 Ufuk Celebi : > > > The Flink PMC is pleased to announce the availability of Flink 1

Re: Flink gives incorrect result when event time windowing used

2016-12-23 Thread Aljoscha Krettek
xOutOfOrderness; > if(potentialWM >= lastEmittedWatermark) { > lastEmittedWatermark = potentialWM; > } > return new Watermark(lastEmittedWatermark); > } > > I think those two implementation should use the same principle. > > > Aljoscha Krettek-2 wrote > > I'm afraid

Re: clean up jira

2016-12-21 Thread Aljoscha Krettek
Thanks a lot for going through the issues and preparing this list! >From a first glance some can definitely be closed. I didn't yet find the time to look through all of them but we should definitely work and cleaning up our Jira. Cheers, Aljoscha On Wed, 21 Dec 2016 at 18:19 Anton Solovev wrote

[jira] [Created] (FLINK-5374) Extend Unit Tests for RegisteredBackendStateMetaInfo

2016-12-20 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5374: --- Summary: Extend Unit Tests for RegisteredBackendStateMetaInfo Key: FLINK-5374 URL: https://issues.apache.org/jira/browse/FLINK-5374 Project: Flink

[jira] [Created] (FLINK-5373) Extend Unit Tests for StateAssignmentOperation

2016-12-20 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5373: --- Summary: Extend Unit Tests for StateAssignmentOperation Key: FLINK-5373 URL: https://issues.apache.org/jira/browse/FLINK-5373 Project: Flink Issue

Re: Flink gives incorrect result when event time windowing used

2016-12-20 Thread Aljoscha Krettek
I'm afraid the doc is wrong here. The JavaDoc on Watermark says this about watermarks: "A Watermark tells operators that receive it that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator." The system also relies on this fact, as visible in how ti

[jira] [Created] (FLINK-5372) Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints()

2016-12-20 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5372: --- Summary: Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints() Key: FLINK-5372 URL: https://issues.apache.org/jira/browse/FLINK-5372 Project: Flink

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-20 Thread Aljoscha Krettek
:55 PM, Fabian Hueske > wrote: > > > >> Hi, > >> > >> I merged the Table API refactoring changes: > >> > >> - RESOLVED Clean up the packages of the Table API (FLINK-4704) > >> - RESOLVED Move Row to flink-core (FLINK-5186) >

[jira] [Created] (FLINK-5366) Add end-to-end tests for Savepoint Backwards Compatibility

2016-12-19 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5366: --- Summary: Add end-to-end tests for Savepoint Backwards Compatibility Key: FLINK-5366 URL: https://issues.apache.org/jira/browse/FLINK-5366 Project: Flink

[jira] [Created] (FLINK-5363) Fire timers when window state is currently empty

2016-12-18 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5363: --- Summary: Fire timers when window state is currently empty Key: FLINK-5363 URL: https://issues.apache.org/jira/browse/FLINK-5363 Project: Flink Issue

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-16 Thread Aljoscha Krettek
Yes, I'm confident that we can finish the tests until then and merge the code. On Fri, Dec 16, 2016, 17:41 Robert Metzger wrote: > Thank you for the update. Do you think you get it done until Monday > evening? > > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek > wrote:

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-12-16 Thread Aljoscha Krettek
Hi, we're still working on making the backwards compatibility from 1.1 savepoints a reality. We have most of the code and some tests now but it still needs some work. This is the issue that tracks the progress on the operators that we would like to make backwards compatible: https://issues.apache.o

Re: trouble using maven to deploy master branch to a private repo

2016-12-12 Thread Aljoscha Krettek
Hi Steve, I think this part of create_release_files.sh (our release script) is helpful: mvn clean deploy -Prelease,docs-and-source --settings deploysettings.xml -DskipTests -Dgpg.executable=$GPG -Dgpg.keyname=$GPG_KEY -Dgpg.passphrase=$GPG_PASSPHRASE -DretryFailedDeploymentCount=10 I think -Prele

[jira] [Created] (FLINK-5250) Make AbstractUdfStreamOperator aware of WrappingFunction

2016-12-04 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5250: --- Summary: Make AbstractUdfStreamOperator aware of WrappingFunction Key: FLINK-5250 URL: https://issues.apache.org/jira/browse/FLINK-5250 Project: Flink

[jira] [Created] (FLINK-5240) Properly Close StateBackend in StreamTask when closing/canceling

2016-12-02 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5240: --- Summary: Properly Close StateBackend in StreamTask when closing/canceling Key: FLINK-5240 URL: https://issues.apache.org/jira/browse/FLINK-5240 Project: Flink

[jira] [Created] (FLINK-5237) Consolidate and harmonize Window Translation Tests

2016-12-02 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5237: --- Summary: Consolidate and harmonize Window Translation Tests Key: FLINK-5237 URL: https://issues.apache.org/jira/browse/FLINK-5237 Project: Flink Issue

[jira] [Created] (FLINK-5181) Add Tests in StateBackendTestBase that verify Default-Value Behaviour

2016-11-28 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5181: --- Summary: Add Tests in StateBackendTestBase that verify Default-Value Behaviour Key: FLINK-5181 URL: https://issues.apache.org/jira/browse/FLINK-5181 Project

Re: Move Row, RowInputFormat to core package

2016-11-28 Thread Aljoscha Krettek
If we move it to core, we have to untangle it from Scala, as Timo said. The reason is that we would like to remove Scala from any user facing API maven packages and if we had it in core everyone would have to suffix maven packages with the Scala version. On Fri, 25 Nov 2016 at 16:47 Anton Solovev

Re: Window's Checkpoint problem

2016-11-28 Thread Aljoscha Krettek
Hi, this is indeed a bug (though I would see it more as a feature since I think using the Checkpointed interface there can indeed be problematic, as Till pointed out). The problem is that the Scala Wrapper functions have to implement all kinds of interfaces so that they can forward to the wrapped f

[jira] [Created] (FLINK-5155) Deprecate ValueStateDescriptor constructors with default value

2016-11-24 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5155: --- Summary: Deprecate ValueStateDescriptor constructors with default value Key: FLINK-5155 URL: https://issues.apache.org/jira/browse/FLINK-5155 Project: Flink

[jira] [Created] (FLINK-5154) Duplicate TypeSerializer when writing RocksDB Snapshot

2016-11-24 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5154: --- Summary: Duplicate TypeSerializer when writing RocksDB Snapshot Key: FLINK-5154 URL: https://issues.apache.org/jira/browse/FLINK-5154 Project: Flink

Re: [DISCUSS] @Public libraries

2016-11-23 Thread Aljoscha Krettek
I would be for also annotating library methods/classes. Maybe Robert has a stronger opinion on this because he introduced these annotations. On Tue, 22 Nov 2016 at 18:56 Greg Hogan wrote: > Hi all, > > Should stable APIs in Flink's CEP, ML, and Gelly libraries be annotated > @Public or restricte

Re: [DISCUSS] Hold copies in HeapStateBackend

2016-11-23 Thread Aljoscha Krettek
t;> So much code changes. Can you show us the key changes code for the > > object > > >> copy? > > >> Object reference maybe hold more deep reference, it can be a bomb. > > >> Can we renew a object with its data or direct use kryo for object > > &

Re: [DISCUSS] deprecated function need more detail

2016-11-23 Thread Aljoscha Krettek
+1 That sounds excellent. On Wed, 23 Nov 2016 at 11:04 Till Rohrmann wrote: > +1 for your proposal. > > Cheers, > Till > > On Wed, Nov 23, 2016 at 9:33 AM, Fabian Hueske wrote: > > > I agree on this one. > > Whenever we deprecate a method or a feature we should add a comment that > > explains t

Re: [DISCUSS] Hold copies in HeapStateBackend

2016-11-23 Thread Aljoscha Krettek
data or direct use kryo for object > > serialization? > > Iā€™m not prefer object copy. > > > > > > > On Nov 22, 2016, at 20:33, Fabian Hueske wrote: > > > > > > Does anybody have objections against copying the first record that goes > > >

[jira] [Created] (FLINK-5130) Remove Deprecated Methods from WindowedStream

2016-11-22 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5130: --- Summary: Remove Deprecated Methods from WindowedStream Key: FLINK-5130 URL: https://issues.apache.org/jira/browse/FLINK-5130 Project: Flink Issue Type

[jira] [Created] (FLINK-5126) Remove Checked Exceptions from State Interfaces

2016-11-22 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5126: --- Summary: Remove Checked Exceptions from State Interfaces Key: FLINK-5126 URL: https://issues.apache.org/jira/browse/FLINK-5126 Project: Flink Issue

[jira] [Created] (FLINK-5125) ContinuousFileProcessingCheckpointITCase is Flaky

2016-11-22 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5125: --- Summary: ContinuousFileProcessingCheckpointITCase is Flaky Key: FLINK-5125 URL: https://issues.apache.org/jira/browse/FLINK-5125 Project: Flink Issue

Re: [DISCUSS] Hold copies in HeapStateBackend

2016-11-22 Thread Aljoscha Krettek
That's right, yes. On Mon, 21 Nov 2016 at 19:14 Fabian Hueske wrote: > Right, but that would be a much bigger change than "just" copying the > *first* record that goes into the ReduceState, or am I missing something? > > > 2016-11-21 18:41 GMT+01:00 Aljoscha Kr

Re: [DISCUSS] Hold copies in HeapStateBackend

2016-11-21 Thread Aljoscha Krettek
en updating the state, but I > > > think it'll be possible to perform asynchronous snapshots using > > > HeapStateBackend (probably some changes to underlying data structures > > would > > > be needed) - which would bring more predictable performance. > &

Re: [DISCUSS] Hold copies in HeapStateBackend

2016-11-21 Thread Aljoscha Krettek
Hi, I would be in favour of this since it brings things in line with the RocksDB backend. This will, however, come with quite the performance overhead, depending on how fast the TypeSerializer can copy. Cheers, Aljoscha On Mon, 21 Nov 2016 at 11:30 Fabian Hueske wrote: > Hi everybody, > > when

[jira] [Created] (FLINK-5061) Remove ContinuousEventTimeTrigger

2016-11-14 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5061: --- Summary: Remove ContinuousEventTimeTrigger Key: FLINK-5061 URL: https://issues.apache.org/jira/browse/FLINK-5061 Project: Flink Issue Type

Re: RocksDB IO error

2016-11-11 Thread Aljoscha Krettek
Hi Zhenhao, does this happen reproducibly? What happens after the failure? Will it retry restoring and then succeed? I have a suspicion that Yarn could be cleaning up some files that RocksDB expects to be there while restoring. Cheers, Aljoscha On Thu, 10 Nov 2016 at 14:24 wrote: > Hi team, >

[DISCUSS] Changing Window Cleanup Semantics

2016-11-09 Thread Aljoscha Krettek
Hi, I recently created https://issues.apache.org/jira/browse/FLINK-4994 to address what I think is a flaw in the window cleanup semantics. This has the possibility of affecting people so I'd like to get some opinions and also give people a heads-up. Before going into what I'm proposing in the issu

[jira] [Created] (FLINK-5037) Instability in AbstractUdfStreamOperatorLifecycleTest

2016-11-09 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5037: --- Summary: Instability in AbstractUdfStreamOperatorLifecycleTest Key: FLINK-5037 URL: https://issues.apache.org/jira/browse/FLINK-5037 Project: Flink

[jira] [Created] (FLINK-5035) Don't Write TypeSerializer to Heap State Snapshot

2016-11-08 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5035: --- Summary: Don't Write TypeSerializer to Heap State Snapshot Key: FLINK-5035 URL: https://issues.apache.org/jira/browse/FLINK-5035 Project: Flink

[jira] [Created] (FLINK-5034) Don't Write StateDescriptor to RocksDB Snapshot

2016-11-08 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5034: --- Summary: Don't Write StateDescriptor to RocksDB Snapshot Key: FLINK-5034 URL: https://issues.apache.org/jira/browse/FLINK-5034 Project: Flink

Re: [DISCUSS] Proposed updates to Flink project site

2016-11-08 Thread Aljoscha Krettek
Hi Mike, I like your proposal. It correctly tackles two things that beginners will appreciate: clearer structure and information about companies that are using Flink. Especially that last part is important because it's easier to adopt a new piece of software if you know that other big players are a

[jira] [Created] (FLINK-5026) Rename TimelyFlatMap to Process

2016-11-07 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5026: --- Summary: Rename TimelyFlatMap to Process Key: FLINK-5026 URL: https://issues.apache.org/jira/browse/FLINK-5026 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-5015) Add Tests/ITCase for Kafka Per-Partition Watermarks

2016-11-04 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5015: --- Summary: Add Tests/ITCase for Kafka Per-Partition Watermarks Key: FLINK-5015 URL: https://issues.apache.org/jira/browse/FLINK-5015 Project: Flink

[jira] [Created] (FLINK-5012) Provide Timestamp in TimelyFlatMapFunction

2016-11-04 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5012: --- Summary: Provide Timestamp in TimelyFlatMapFunction Key: FLINK-5012 URL: https://issues.apache.org/jira/browse/FLINK-5012 Project: Flink Issue Type

[jira] [Created] (FLINK-5003) Provide Access to State Stores in Operator Snapshot Context

2016-11-03 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5003: --- Summary: Provide Access to State Stores in Operator Snapshot Context Key: FLINK-5003 URL: https://issues.apache.org/jira/browse/FLINK-5003 Project: Flink

[jira] [Created] (FLINK-5000) Rename Methods in ManagedInitializationContext

2016-11-02 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-5000: --- Summary: Rename Methods in ManagedInitializationContext Key: FLINK-5000 URL: https://issues.apache.org/jira/browse/FLINK-5000 Project: Flink Issue

[jira] [Created] (FLINK-4994) Don't Clear Trigger State and Merging Window Set When Purging

2016-11-02 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4994: --- Summary: Don't Clear Trigger State and Merging Window Set When Purging Key: FLINK-4994 URL: https://issues.apache.org/jira/browse/FLINK-4994 Project:

Re: [DISCUSS] FLIP-2 Extending Window Function Metadata

2016-11-02 Thread Aljoscha Krettek
nu Zhang wrote: > Hi Aljoscha, > > Have you started working on ProcessWindowFunction ? If not, may I take this > task ? > > Thanks, > Manu > > > On Wed, Nov 2, 2016 at 5:16 PM Aljoscha Krettek > wrote: > > > I think we reached consensus here so I would li

[jira] [Created] (FLINK-4993) Don't Allow Trigger.onMerge() to return TriggerResult

2016-11-02 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4993: --- Summary: Don't Allow Trigger.onMerge() to return TriggerResult Key: FLINK-4993 URL: https://issues.apache.org/jira/browse/FLINK-4993 Project:

Re: [DISCUSS] FLIP-2 Extending Window Function Metadata

2016-11-02 Thread Aljoscha Krettek
I think we reached consensus here so I would like to mark this FLIP as accepted. We will now process with implementing the first step, i.e. adding the new ProcessWindowFunction. On Mon, 1 Aug 2016 at 18:08 Aljoscha Krettek wrote: > Alright, that seems reasonable. I updated the doc to add

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-11-01 Thread Aljoscha Krettek
Thanks, I'm having a look the the PR right now. On Tue, 1 Nov 2016 at 04:57 Vishnu Viswanath wrote: > Hi, > > I have created a pull request for this: > https://github.com/apache/flink/pull/2736 > > Regards, > Vishnu > > On Tue, Oct 18, 2016 at 3:34 AM, Alj

Re: [DISCUSSION] Timely function interface and timer params

2016-11-01 Thread Aljoscha Krettek
amespaces, even if it's simpler > > and just a string, is the way to go. > > > > I'm really excited by this guys! I think the TimelyFlatMap and > > TimelyCoFlatMap are going to get a LOT of use. This is gonna make a lot > of > > people happy. > > >

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

2016-11-01 Thread Aljoscha Krettek
Hi, thanks for trying to revive the discussion! I added some comments in the doc. Cheers, Aljoscha On Fri, 28 Oct 2016 at 12:05 venturadelmonte wrote: > Hello, > > I find this feature really cool because it would allow people to tackle > scenarios requiring a more advanced "join" on multiple st

Re: [DISCUSSION] Timely function interface and timer params

2016-10-28 Thread Aljoscha Krettek
Hi Gyula, if you look at the internal API you'll notice that it is pretty much like your second proposal. Just for reference, the interface is roughly this: public interface InternalTimerService { long currentProcessingTime(); long currentWatermark(); void registerProcessingTimeTimer(N names

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-10-28 Thread Aljoscha Krettek
> > > +1 for the features to include. > > > > What is the state of the Trigger DSL? How much is left to be done before > > merging? > > > > Cheers, > > Till > > > > On Tue, Oct 25, 2016 at 4:32 PM, Aljoscha Krettek > > wrote: > >

[jira] [Created] (FLINK-4959) Write Documentation for Timely FlatMap Functions

2016-10-28 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4959: --- Summary: Write Documentation for Timely FlatMap Functions Key: FLINK-4959 URL: https://issues.apache.org/jira/browse/FLINK-4959 Project: Flink Issue

[jira] [Created] (FLINK-4957) Provide API for TimelyCoFlatMapFunction

2016-10-28 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4957: --- Summary: Provide API for TimelyCoFlatMapFunction Key: FLINK-4957 URL: https://issues.apache.org/jira/browse/FLINK-4957 Project: Flink Issue Type: Test

[jira] [Created] (FLINK-4955) Add Translations Tests for KeyedStream.flatMap(TimelyFlatMapFunction)

2016-10-28 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4955: --- Summary: Add Translations Tests for KeyedStream.flatMap(TimelyFlatMapFunction) Key: FLINK-4955 URL: https://issues.apache.org/jira/browse/FLINK-4955 Project

[jira] [Created] (FLINK-4940) Add Support for Broadcast/Global State

2016-10-27 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4940: --- Summary: Add Support for Broadcast/Global State Key: FLINK-4940 URL: https://issues.apache.org/jira/browse/FLINK-4940 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-4924) Simplify Operator Test Harness Constructors

2016-10-26 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4924: --- Summary: Simplify Operator Test Harness Constructors Key: FLINK-4924 URL: https://issues.apache.org/jira/browse/FLINK-4924 Project: Flink Issue Type

Re: [DISCUSS] Schedule and Scope for Flink 1.2

2016-10-25 Thread Aljoscha Krettek
+1 the schedule proposed so far. Do we also want to get in the "Trigger DSL" that we've had brewing for a while now? On Mon, 17 Oct 2016 at 16:17 Stephan Ewen wrote: > I think this sounds very reasonable, +1 to the schedule. > > I would definitely add > - FLIP-10 (unify checkpoints and savepo

[jira] [Created] (FLINK-4907) Add Test for Timers/State Provided by AbstractStreamOperator

2016-10-25 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4907: --- Summary: Add Test for Timers/State Provided by AbstractStreamOperator Key: FLINK-4907 URL: https://issues.apache.org/jira/browse/FLINK-4907 Project: Flink

Re: [FLINK-3035] Redis as State Backend

2016-10-24 Thread Aljoscha Krettek
or's state in hdfs I assume > that on failure you are > restarting the operator's tasks on other nodes, is that possible with > RocksDB? > > Best, > Ovidiu > > -Original Message- > From: Aljoscha Krettek [mailto:aljos...@apache.org] > Sent: Monday, Octobe

Re: [FLINK-4832] Count/Sum 0 elements

2016-10-24 Thread Aljoscha Krettek
Hi Anton, executeOnCollection() is only meant for executing Flink Jobs in the local machine without bringing up a local (or actual) Flink cluster. So solving the problem there does not really solve the problem. The underlying problem is this: in a Map-Reduce world the way to count elements of type

[jira] [Created] (FLINK-4892) Snapshot TimerService using Key-Grouped State

2016-10-24 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4892: --- Summary: Snapshot TimerService using Key-Grouped State Key: FLINK-4892 URL: https://issues.apache.org/jira/browse/FLINK-4892 Project: Flink Issue Type

[jira] [Created] (FLINK-4884) Eagerly Store MergingWindowSet in State in WindowOperator

2016-10-21 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4884: --- Summary: Eagerly Store MergingWindowSet in State in WindowOperator Key: FLINK-4884 URL: https://issues.apache.org/jira/browse/FLINK-4884 Project: Flink

Re: TopSpeedWindowing - in error: Could not forward element to next operator

2016-10-21 Thread Aljoscha Krettek
Hi, the problem is that EvictingWindowOperator uses StreamRecordSerializer to serialise the contents of the windows. This does not serialise timestamps so when the objects are deserialised from RocksDB they all have Long.MIN_VALUE as timestamp. The evictor in the program therefore always evicts all

[jira] [Created] (FLINK-4877) Refactorings around FLINK-3674 (User Function Timers)

2016-10-21 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4877: --- Summary: Refactorings around FLINK-3674 (User Function Timers) Key: FLINK-4877 URL: https://issues.apache.org/jira/browse/FLINK-4877 Project: Flink

[jira] [Created] (FLINK-4866) Make Trigger.clear() Abstract to Enforce Implementation

2016-10-20 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4866: --- Summary: Make Trigger.clear() Abstract to Enforce Implementation Key: FLINK-4866 URL: https://issues.apache.org/jira/browse/FLINK-4866 Project: Flink

[jira] [Created] (FLINK-4859) Clearly Separate Responsibilities of StreamOperator and StreamTask

2016-10-19 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4859: --- Summary: Clearly Separate Responsibilities of StreamOperator and StreamTask Key: FLINK-4859 URL: https://issues.apache.org/jira/browse/FLINK-4859 Project

[jira] [Created] (FLINK-4858) Remove Legacy Checkpointing Interfaces

2016-10-19 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4858: --- Summary: Remove Legacy Checkpointing Interfaces Key: FLINK-4858 URL: https://issues.apache.org/jira/browse/FLINK-4858 Project: Flink Issue Type

Re: Add MapState for keyed streams

2016-10-19 Thread Aljoscha Krettek
Perfect! Then it's pretty much what we discussed here: https://issues.apache.org/jira/browse/FLINK-3947 and I'm very much in favour of that. Just the implementation of RocksDB could be a bit tricky but it should be doable. Cheers, Aljoscha On Wed, 19 Oct 2016 at 11:43 Jark Wu wrote: > Hi Xiaoga

Re: Add MapState for keyed streams

2016-10-19 Thread Aljoscha Krettek
Hi, just making sure I understand this correctly. Would the MapState keys be the same keys as the one provided when creating the KeyedStream or a different key. As an example, would it be like this: DataStream> input = ...; KeyedStream keyed = input.keyBy(0) keyed.map( Tuple2 input -> mapState.pu

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-10-18 Thread Aljoscha Krettek
another class for holding (timestamp,value) > tuple? > > Regards, > Vishnu > > On Mon, Oct 17, 2016 at 4:19 AM, Aljoscha Krettek > wrote: > > > Hi Vishnu, > > what you suggested is spot on! Please go forward with it like this. > > > > One small sugge

Re: Flink Metrics

2016-10-18 Thread Aljoscha Krettek
https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html Or this: https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html if you prefer Flink 1.1 On Mon, 17 Oct 2016 at 19:16 amir bahmanyari wrote: > Hi colleagues, > Is there a link that described

Re: [FLINK-3035] Redis as State Backend

2016-10-17 Thread Aljoscha Krettek
s that we have to somehow get at the > state in redis for checkpointing. And if we use only one Redis instance for > all states then this will be problematic." - Aljoscha Krettek > > Any other update on this issue will help, not clear the status. > > Best, > Ovidiu > >

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-10-17 Thread Aljoscha Krettek
; > > vishnu.viswanat...@gmail.com> wrote: > > > > > Thank you Aljoscha, > > > > > > Yes, I agree we don't need ProcessingTimeEvcitor. > > > I will change the current TimeEvictors to use EventTimeEvictor as > > > suggested. > > > >

Re: [DISCUSS] Removing delete*Timer from the WindowOperator.Context

2016-10-11 Thread Aljoscha Krettek
+Konstantin Knauf looping you in directly because you used the "delete timer" feature in the past and even did some changes to the timer system. Are you still relying on the fact that deleted timers are actually deleted. The main reason for wanting to get rid of delete timer is IMHO that deleting

[DISCUSS] Timely User Functions and Watermarks

2016-09-27 Thread Aljoscha Krettek
Hi Folks, I'm in the process of implementing https://issues.apache.org/jira/browse/FLINK-3674 and now I'm having a bit of a problem with deciding how watermarks should be treated for operators that have more than one input. The problem is deciding when to fire event-time timers. For one-input oper

Re: No support for request PutMappingRequest

2016-09-27 Thread Aljoscha Krettek
.startObject("raw") > .field("type","string") > .field("index", "not_analyzed") > .endObject() > .endObject() > .endObject() > >

Re: No support for request PutMappingRequest

2016-09-26 Thread Aljoscha Krettek
switched to FAILED > > > We try to add mapping request to elastic search. We cannot access to > client attribute (it is private) in elasticsearch class. > > > Is there any way to overcome this problem. > > > Thanks, > > > Ozan > > > > __

[jira] [Created] (FLINK-4675) Remove Parameter from WindowAssigner.getDefaultTrigger()

2016-09-24 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4675: --- Summary: Remove Parameter from WindowAssigner.getDefaultTrigger() Key: FLINK-4675 URL: https://issues.apache.org/jira/browse/FLINK-4675 Project: Flink

Re: [DISCUSS] Merge batch and stream connector modules

2016-09-23 Thread Aljoscha Krettek
I don't think it's that easy. The streaming connectors have flink-streaming as dependency while the batch connectors have the batch dependencies. Combining them would mean that users always have all dependencies, right? On Thu, 22 Sep 2016 at 15:41 Stephan Ewen wrote: > +1 for Fabian's suggesti

Re: Get Flink ExecutionGraph Programmatically

2016-09-22 Thread Aljoscha Krettek
Hi, there is ClusterClient.getAccumulators(JobID jobID) which should be able to get the accumulators for a running job. If you can construct a ClusterClient that should be a good solution. Cheers, Aljoscha On Wed, 21 Sep 2016 at 21:15 Chawla,Sumit wrote: > Hi Sean > > My goal here is to get Use

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-09-20 Thread Aljoscha Krettek
eparate it from this FLIP > and create a JIRA for it, what do you say? > > Please let me know your thoughts. > > Regards, > Vishnu > > On Sun, Jul 31, 2016 at 12:07 PM, Aljoscha Krettek > wrote: > > > Hi, > > regarding a), b) and c): The WindowOperato

Re: Implementing FLIP-2 and FLIP-4

2016-09-17 Thread Aljoscha Krettek
lly), and I would very much like to > work on this now if I can justify it. If not, I would still very much like > to work on this, but the timing will have to be different. > > Again, thank you Aljoscha, and I apologize for the rushed nature of my > situation. > > Best, >

Re: checkpointing a OneInputStreamOperator/AbstractStreamOperator

2016-09-14 Thread Aljoscha Krettek
? > > Dan > > On Mon, Sep 12, 2016 at 11:33 PM Aljoscha Krettek > wrote: > > > Hi, > > yes you guessed correctly: CheckpointedAsynchronously only works with > > functions and not with the lower-level StreamOperator. You would have to > > implement snapshotOp

Re: Implementing FLIP-2 and FLIP-4

2016-09-14 Thread Aljoscha Krettek
Hi AJ, the idea for evictors initially came from IBM Infosphere Streams, if I'm not mistaken: http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.dev.doc/doc/windowhandling.html The first version of the windowing system used a combination of triggers/evictors to do the windowing

Re: checkpointing a OneInputStreamOperator/AbstractStreamOperator

2016-09-12 Thread Aljoscha Krettek
Hi, yes you guessed correctly: CheckpointedAsynchronously only works with functions and not with the lower-level StreamOperator. You would have to implement snapshotOperatorState() and restoreState(). These interfaces are quite low-level, though, and not stable. For example, in Flink 1.2 we're refa

[jira] [Created] (FLINK-4602) Move RocksDB backed to proper package

2016-09-09 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4602: --- Summary: Move RocksDB backed to proper package Key: FLINK-4602 URL: https://issues.apache.org/jira/browse/FLINK-4602 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-4589) Fix Merging of Covering Window in MergingWindowSet

2016-09-07 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4589: --- Summary: Fix Merging of Covering Window in MergingWindowSet Key: FLINK-4589 URL: https://issues.apache.org/jira/browse/FLINK-4589 Project: Flink Issue

[jira] [Created] (FLINK-4588) Fix Merging of Covering Window in MergingWindowSet

2016-09-07 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4588: --- Summary: Fix Merging of Covering Window in MergingWindowSet Key: FLINK-4588 URL: https://issues.apache.org/jira/browse/FLINK-4588 Project: Flink Issue

[jira] [Created] (FLINK-4579) Add StateBackendFactory for RocksDB Backend

2016-09-05 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4579: --- Summary: Add StateBackendFactory for RocksDB Backend Key: FLINK-4579 URL: https://issues.apache.org/jira/browse/FLINK-4579 Project: Flink Issue Type

Re: [VOTE] Release Apache Flink 1.1.2 (RC1)

2016-09-01 Thread Aljoscha Krettek
+1 I went over all the changes that we introduced since 1.1.1 and they look good. On Wed, 31 Aug 2016 at 14:53 Maximilian Michels wrote: > Found a minor bug for detached job submissions but I wouldn't cancel > the release for it: https://issues.apache.org/jira/browse/FLINK-4540 > > On Wed, Aug

[ANNOUNCE] Queryable State is Broken on Master

2016-09-01 Thread Aljoscha Krettek
Our recent changes to make keyed state rescalable/key-group aware are breaking queryable state because it is not yet made key-group aware. I opened this Jira issue to track the fix for that: https://issues.apache.org/jira/browse/FLINK-4556. Sorry for the inconvenience. Cheers, Aljoscha

[jira] [Created] (FLINK-4556) Make Queryable State Key-Group Aware

2016-09-01 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4556: --- Summary: Make Queryable State Key-Group Aware Key: FLINK-4556 URL: https://issues.apache.org/jira/browse/FLINK-4556 Project: Flink Issue Type

Re: Extending FLIP template

2016-09-01 Thread Aljoscha Krettek
+1 If you think it worthwhile you can add it to the template(s). On Thu, 1 Sep 2016 at 10:38 Fabian Hueske wrote: > Hi, > > I'm currently preparing a FLIP for Table API streaming aggregates and > noticed that there is no section about how the task can be divided into > subtasks. > > I think it w

<    5   6   7   8   9   10   11   12   13   14   >