[jira] [Created] (FLINK-9758) ContinuousFileProcessingTest failed due to not setting runtimeContext

2018-07-04 Thread Yun Tang (JIRA)
Yun Tang created FLINK-9758: --- Summary: ContinuousFileProcessingTest failed due to not setting runtimeContext Key: FLINK-9758 URL: https://issues.apache.org/jira/browse/FLINK-9758 Project: Flink

Re: Consuming data from dynamoDB streams to flink

2018-07-04 Thread Tzu-Li (Gordon) Tai
Hi Ying, Sorry for the late reply here. >From the looks of the AmazonDynamoDBStreamsClient, yes it seems like this should simply work. Regarding the resharding behaviour I mentioned in the JIRA: I'm not sure if this is really a difference in behaviour. Internally, if DynamoDB streams is

[jira] [Created] (FLINK-9757) Typos found in docs after hunspell run

2018-07-04 Thread Sergey Nuyanzin (JIRA)
Sergey Nuyanzin created FLINK-9757: -- Summary: Typos found in docs after hunspell run Key: FLINK-9757 URL: https://issues.apache.org/jira/browse/FLINK-9757 Project: Flink Issue Type:

[jira] [Created] (FLINK-9756) Exceptions in BufferListener#notifyBufferAvailable do not trigger further listeners in LocalBufferPool#recycle()

2018-07-04 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-9756: -- Summary: Exceptions in BufferListener#notifyBufferAvailable do not trigger further listeners in LocalBufferPool#recycle() Key: FLINK-9756 URL:

[jira] [Created] (FLINK-9755) Exceptions in RemoteInputChannel#notifyBufferAvailable() are not propagated to the responsible thread

2018-07-04 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-9755: -- Summary: Exceptions in RemoteInputChannel#notifyBufferAvailable() are not propagated to the responsible thread Key: FLINK-9755 URL: https://issues.apache.org/jira/browse/FLINK-9755

Re: [DISCUSS] Flink 1.6 features

2018-07-04 Thread Stephan Ewen
Hi all! A late follow-up with some thoughts: In principle, all these are good suggestions and are on the roadmap. We are trying to make the release "by time", meaning for it at a certain date (roughly in two weeks) and take what features are ready into the release. Looking at the status of the

Re: [PROPOSAL] Introduce Elastic Bloom Filter For Flink

2018-07-04 Thread Stephan Ewen
Hi Sihua! Sorry for joining this discussion late. I can see the benefit of such a feature and also see the technical merit. It is a nice piece of work and a good proposal. I am wondering if there is a way to add such a technique as a "library operator", or whether it needs a deep integration

[jira] [Created] (FLINK-9754) deploy_staging_jars.sh refers to non-existing profile

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9754: --- Summary: deploy_staging_jars.sh refers to non-existing profile Key: FLINK-9754 URL: https://issues.apache.org/jira/browse/FLINK-9754 Project: Flink

[jira] [Created] (FLINK-9753) Support ORC/Parquet for StreamingFileSink

2018-07-04 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-9753: --- Summary: Support ORC/Parquet for StreamingFileSink Key: FLINK-9753 URL: https://issues.apache.org/jira/browse/FLINK-9753 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-9752) Add an S3 RecoverableWriter

2018-07-04 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-9752: --- Summary: Add an S3 RecoverableWriter Key: FLINK-9752 URL: https://issues.apache.org/jira/browse/FLINK-9752 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-9751) Add a RecoverableWriter to the FileSystem abstraction

2018-07-04 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-9751: --- Summary: Add a RecoverableWriter to the FileSystem abstraction Key: FLINK-9751 URL: https://issues.apache.org/jira/browse/FLINK-9751 Project: Flink Issue

[jira] [Created] (FLINK-9750) Create new StreamingFileSink that works on Flink's FileSystem abstraction

2018-07-04 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-9750: --- Summary: Create new StreamingFileSink that works on Flink's FileSystem abstraction Key: FLINK-9750 URL: https://issues.apache.org/jira/browse/FLINK-9750 Project: Flink

[jira] [Created] (FLINK-9749) Rework Bucketing Sink

2018-07-04 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-9749: --- Summary: Rework Bucketing Sink Key: FLINK-9749 URL: https://issues.apache.org/jira/browse/FLINK-9749 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-9748) create_source_release pollutes flink root directory

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9748: --- Summary: create_source_release pollutes flink root directory Key: FLINK-9748 URL: https://issues.apache.org/jira/browse/FLINK-9748 Project: Flink

[jira] [Created] (FLINK-9747) Release guide ambigous regarding release branches

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9747: --- Summary: Release guide ambigous regarding release branches Key: FLINK-9747 URL: https://issues.apache.org/jira/browse/FLINK-9747 Project: Flink Issue

[jira] [Created] (FLINK-9746) Release guide says to check docs/_config.yml, which is later modified

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9746: --- Summary: Release guide says to check docs/_config.yml, which is later modified Key: FLINK-9746 URL: https://issues.apache.org/jira/browse/FLINK-9746 Project:

[jira] [Created] (FLINK-9745) Release guide misses information for updating version for releases

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9745: --- Summary: Release guide misses information for updating version for releases Key: FLINK-9745 URL: https://issues.apache.org/jira/browse/FLINK-9745 Project:

[jira] [Created] (FLINK-9744) update_branch_version script produces incorrect output for non-shapshot versions

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9744: --- Summary: update_branch_version script produces incorrect output for non-shapshot versions Key: FLINK-9744 URL: https://issues.apache.org/jira/browse/FLINK-9744

Re: Job jumbo jar can't be extracted on Windows

2018-07-04 Thread Chesnay Schepler
https://issues.apache.org/jira/browse/FLINK-9743 On 04.07.2018 16:33, Chesnay Schepler wrote: could reproduce it, sounds about right that we should always replace "/". On 04.07.2018 16:14, NEKRASSOV, ALEXEI wrote: To demonstrate this problem: 1. Get a jar that includes other jar's. For

[jira] [Created] (FLINK-9743) PackagedProgram.extractContainedLibraries fails on Windows

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9743: --- Summary: PackagedProgram.extractContainedLibraries fails on Windows Key: FLINK-9743 URL: https://issues.apache.org/jira/browse/FLINK-9743 Project: Flink

Re: Job jumbo jar can't be extracted on Windows

2018-07-04 Thread Chesnay Schepler
could reproduce it, sounds about right that we should always replace "/". On 04.07.2018 16:14, NEKRASSOV, ALEXEI wrote: To demonstrate this problem: 1. Get a jar that includes other jar's. For example, download

RE: Job jumbo jar can't be extracted on Windows

2018-07-04 Thread NEKRASSOV, ALEXEI
To demonstrate this problem: 1. Get a jar that includes other jar's. For example, download http://central.maven.org/maven2/org/eclipse/platform/org.eclipse.ant.ui/3.6.201/org.eclipse.ant.ui-3.6.201.jar 2. Put downloaded jar in c:\usr\tmp\ 3. In cmd (note that we attempt to create

[jira] [Created] (FLINK-9742) Widen scope of Expression.resultType to 'public'

2018-07-04 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created FLINK-9742: --- Summary: Widen scope of Expression.resultType to 'public' Key: FLINK-9742 URL: https://issues.apache.org/jira/browse/FLINK-9742 Project: Flink Issue Type:

[jira] [Created] (FLINK-9741) Register JVM metrics for each JM separately

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9741: --- Summary: Register JVM metrics for each JM separately Key: FLINK-9741 URL: https://issues.apache.org/jira/browse/FLINK-9741 Project: Flink Issue Type:

[jira] [Created] (FLINK-9740) Support group windows over intervals of months

2018-07-04 Thread Timo Walther (JIRA)
Timo Walther created FLINK-9740: --- Summary: Support group windows over intervals of months Key: FLINK-9740 URL: https://issues.apache.org/jira/browse/FLINK-9740 Project: Flink Issue Type: New

Re: Flink memory management in table api

2018-07-04 Thread Fabian Hueske
State is maintained in the configured state backend [1]. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/state_backends.html 2018-07-04 11:38 GMT+02:00 Amol S - iProgrammer : > Hello fabian, > > Thanks for your quick response, > > According to above

Re: [DISCUSS] Release Flink 1.5.1

2018-07-04 Thread Till Rohrmann
Quick update for FLINK-9676: It is not super critical/blocker to fix this issue for Flink 1.5.1 since the deadlock only occurs in case of a task cancellation and also then we haven't seen it to happen too often. If Flink should deadlock, then there is still the TaskCancelerWatchDog which would

Re: Flink memory management in table api

2018-07-04 Thread Amol S - iProgrammer
Hello fabian, Thanks for your quick response, According to above conversation flink will persist state forever for non windowed operations. I want to know how flink persiat the state i.e. Database or file system or in memory etc. On Wed, 4 Jul 2018 at 2:12 PM, Fabian Hueske wrote: > Hi Amol,

Re: [DISCUSS] Release Flink 1.5.1

2018-07-04 Thread Chesnay Schepler
Alrighty, looks like we're in agreement for a soon release. Let's take inventory of issues to still include: - [FLINK-9554] flink scala shell doesn't work in yarn mode I'm currently taking a look and will probably merge it by the end of today. - [FLINK-9676] Deadlock during canceling task and

Re: Job jumbo jar can't be extracted on Windows

2018-07-04 Thread Chesnay Schepler
Can you provide us with instructions for reproducing this error? Please also include the versions for the jdk/tools/build-system etc. I frequently submit jobs on windows and haven't ran into this issue. On 03.07.2018 20:05, Ted Yu wrote: Looking at javadoc of separatorChar : * The

[jira] [Created] (FLINK-9739) Regression in supported filesystems for RocksDB

2018-07-04 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9739: --- Summary: Regression in supported filesystems for RocksDB Key: FLINK-9739 URL: https://issues.apache.org/jira/browse/FLINK-9739 Project: Flink Issue

Re: Flink memory management in table api

2018-07-04 Thread Fabian Hueske
Hi Amol, The memory consumption depends on the query/operation that you are doing. Time-based operations like group-window-aggregations, over-window-aggregations, or window-joins can automatically clean up their state once data is not no longer needed. Operations such as non-windowed aggregations

[jira] [Created] (FLINK-9738) Provide a way to define Table Version Functions in Table API

2018-07-04 Thread Piotr Nowojski (JIRA)
Piotr Nowojski created FLINK-9738: - Summary: Provide a way to define Table Version Functions in Table API Key: FLINK-9738 URL: https://issues.apache.org/jira/browse/FLINK-9738 Project: Flink

[jira] [Created] (FLINK-9737) Allow to define Table Version Functions from SQL environment file

2018-07-04 Thread Piotr Nowojski (JIRA)
Piotr Nowojski created FLINK-9737: - Summary: Allow to define Table Version Functions from SQL environment file Key: FLINK-9737 URL: https://issues.apache.org/jira/browse/FLINK-9737 Project: Flink

[ANNOUNCE] Feature freeze date for Flink 1.6

2018-07-04 Thread Till Rohrmann
Dear community, as discussed in this thread [1], we will freeze the feature development for Flink 1.6 on the 15th of July 23:59 CET. Please try to finish you feature work until then. If this is not possible, then don't worry because it will then be released with Flink 1.7+. [1]

Re: [DISCUSS] Feature freeze for Flink 1.6

2018-07-04 Thread Till Rohrmann
I assume that we reached a silent consensus here. I will then announce the feature freeze date on the dev mailing list. Cheers, Till On Fri, Jun 29, 2018 at 6:31 PM Till Rohrmann wrote: > Dear community, > > as discussed in the Flink 1.6 feature thread [1], the proposed release > date of *end

[jira] [Created] (FLINK-9736) Potential null reference in KeyGroupPartitionedPriorityQueue#poll()

2018-07-04 Thread Ted Yu (JIRA)
Ted Yu created FLINK-9736: - Summary: Potential null reference in KeyGroupPartitionedPriorityQueue#poll() Key: FLINK-9736 URL: https://issues.apache.org/jira/browse/FLINK-9736 Project: Flink Issue

[jira] [Created] (FLINK-9735) Potential resource leak in RocksDBStateBackend#getDbOptions

2018-07-04 Thread Ted Yu (JIRA)
Ted Yu created FLINK-9735: - Summary: Potential resource leak in RocksDBStateBackend#getDbOptions Key: FLINK-9735 URL: https://issues.apache.org/jira/browse/FLINK-9735 Project: Flink Issue Type: Bug