Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

It seems that we have done that (?). The BatchTableEnvironment.registerTableSource(name, tableSource) only accept a BatchTableSource. In contrast, the StreamTableEnvironment.registerTableSource(name, tableSource) only accept a StreamTableSource. So that, if a TableSource implements both batch

Re: Memory Management in Streaming mode

Hi Shen, What you've read in the documentation is correct. The managed memory is only used by the batch operators. For streaming, we have different state backends with different characteristics. The FsStateBackend keeps the state (including window contents) on the heap and makes backups to the

[jira] [Created] (FLINK-4531) Thread.stop() is deprecated

Ted Yu created FLINK-4531: - Summary: Thread.stop() is deprecated Key: FLINK-4531 URL: https://issues.apache.org/jira/browse/FLINK-4531 Project: Flink Issue Type: Bug Reporter: Ted Yu

Re: Reducing the JIRA PR message verbosity

> Will work log updates still lead to notifications on the mailing list? I don't know but it is most likely configurable. I would prefer if we didn't get notifications for work log items because these would again be duplicates of the GitHub notifications. On Mon, Aug 29, 2016 at 5:39 PM, Neelesh

Re: why job submit timeout is 21474835 second

This limitation doesn't exist anymore in the latest master. Jobs may be monitored for infinite amount of time now. Note that it wouldn't cancel the job if the submission timeout had been reached before the job completed. On Mon, Aug 29, 2016 at 6:06 PM, Till Rohrmann wrote:

Re: why job submit timeout is 21474835 second

If I'm not mistaken then this was a limitation of Akka's ask futures. The maximum timeout was as specified. It might have changed with a new version, though. Usually, you should rarely run into a case where this actually matters. Cheers, Till On Mon, Aug 29, 2016 at 5:44 PM, Greg Hogan

why job submit timeout is 21474835 second

AkkaUtils.scala val INF_TIMEOUT = 21474835 seconds That is job submit timeout 248.55 days. Why is this number?

Memory Management in Streaming mode

Hi, I have a question regarding memory management in the Streaming mode. The documents state that the memory management module is only used in the Batch mode, and Flink Streaming directly operates on JVM heap. Then, what if the volume of data in a window becomes too large to fit in the heap? Will

Re: Reducing the JIRA PR message verbosity

Thanks Max for coordinating the effort. +1 as well. On Mon, Aug 29, 2016 at 6:57 AM, Robert Metzger wrote: > +1 > > I didn't know that we can put the comments into the work log. > Will work log updates still lead to notifications on the mailing list? > > On Mon, Aug 29,

Re: why job submit timeout is 21474835 second

Could be rewritten as "val INFO_TIMEOUT = Integer.MAX_VALUE seconds"? On Mon, Aug 29, 2016 at 4:22 AM, 时金魁 wrote: > > > AkkaUtils.scala > val INF_TIMEOUT = 21474835 seconds > > > That is job submit timeout 248.55 days. > > > Why is this number? > > > >

[jira] [Created] (FLINK-4530) Generalize TaskExecutorToResourceManagerConnection to be reusable

Till Rohrmann created FLINK-4530: Summary: Generalize TaskExecutorToResourceManagerConnection to be reusable Key: FLINK-4530 URL: https://issues.apache.org/jira/browse/FLINK-4530 Project: Flink

[jira] [Created] (FLINK-4529) Move JobMaster, TaskExecutor and ResourceManager out of the rpc package

Till Rohrmann created FLINK-4529: Summary: Move JobMaster, TaskExecutor and ResourceManager out of the rpc package Key: FLINK-4529 URL: https://issues.apache.org/jira/browse/FLINK-4529 Project: Flink

Scheduling of Flink jobs

Hi, I am running Flink on a cluster of 5 nodes. Here is my config: *taskmanager.numberOfTaskSlots: 1parallelism.default: 1* My Flink dashboard shows the following: *Task Managers: 5* *Task Slots: 5* *Available Task Slots: 5* I have the following questions: 1. Why does a job with 8 tasks

Re: Reducing the JIRA PR message verbosity

+1 I didn't know that we can put the comments into the work log. Will work log updates still lead to notifications on the mailing list? On Mon, Aug 29, 2016 at 11:52 AM, Maximilian Michels wrote: > From what I understand so far, the message mirroring can be adjusted > in the

[jira] [Created] (FLINK-4528) Mark runAsync and scheduleAsync methods protected

Till Rohrmann created FLINK-4528: Summary: Mark runAsync and scheduleAsync methods protected Key: FLINK-4528 URL: https://issues.apache.org/jira/browse/FLINK-4528 Project: Flink Issue Type:

[jira] [Created] (FLINK-4527) Drop the "flinkAccumulators" from the Execution

Stephan Ewen created FLINK-4527: --- Summary: Drop the "flinkAccumulators" from the Execution Key: FLINK-4527 URL: https://issues.apache.org/jira/browse/FLINK-4527 Project: Flink Issue Type:

[jira] [Created] (FLINK-4526) ApplicationClient: remove redundant proxy messages

Maximilian Michels created FLINK-4526: - Summary: ApplicationClient: remove redundant proxy messages Key: FLINK-4526 URL: https://issues.apache.org/jira/browse/FLINK-4526 Project: Flink

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

At first glance, I thought we are losing the possibility to distingish between choosing a batch or streaming table if a TableSource implements both. Because currently you are using a StreamTableSource as default if a TableSource implements both types. I think it would be better to determine

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Timo, Yes, you are right. The STREAM keyword is invalid now. If there is a STREAM keyword in the query, the parser will throw "can’t convert table xxx to stream" exception. Because we register the table as a regular table not streamable. - Jark Wu > 在 2016年8月29日，下午8:13，Timo Walther

Re: [DISCUSS] Some thoughts about unify Stream SQL and Batch SQL grammer

Hi Jark, your code looks good and it also simplifies many parts. So the STREAM keyword is not optional but invalid now, right? What happens if there is keyword in the query? Timo Am 29/08/16 um 05:40 schrieb Jark Wu: Hi Fabian, Timo, I have created a prototype for removing STREAM keyword

[jira] [Created] (FLINK-4525) Drop the "eager split pre-assignment" code paths

Stephan Ewen created FLINK-4525: --- Summary: Drop the "eager split pre-assignment" code paths Key: FLINK-4525 URL: https://issues.apache.org/jira/browse/FLINK-4525 Project: Flink Issue Type:

Re: Reducing the JIRA PR message verbosity

>From what I understand so far, the message mirroring can be adjusted in the follow parts: 1) GitHub PR description 2) GitHub Diff comments 3) GitHub Main thread comments We can choose to put these as 1) The full GitHub comment 2) Only supply a link to the GitHub comment Next, we can store all

Re: [DISCUSS] Releasing Flink 1.1.2

Gordon fixed the Kinesis consumer already over the weekend. I just merged the change. I'll push a hotfix for the jodatime+quickstart issue to both branches. On Fri, Aug 26, 2016 at 6:35 PM, Tzu-Li (Gordon) Tai wrote: > Next week Monday evening should be fine to get the

Re: Yarn node went down...

Thanks for reporting Niels. We'll look into it ASAP. On Mon, Aug 29, 2016 at 11:33 AM, Maximilian Michels wrote: > Danke für's Weiterleiten. > > On Mon, Aug 29, 2016 at 11:25 AM, Till Rohrmann > wrote: >> For the attention of the YARN shepherd.

Re: Yarn node went down...

Thanks for reporting Niels. We'll look into it ASAP. On Mon, Aug 29, 2016 at 10:31 AM, Niels Basjes wrote: > Hi, > > Last week I brought down one of our Yarn nodes because of this problem: > https://issues.apache.org/jira/browse/FLINK-4485 > > The Yarn node no longer accepted

Re: Check-pointing configuration

Hi! Right now, you would have to activate it on the StreamExecutionEnvironment. We did not at a global setting for that, because each job may choose different settings (like checkpointing interval). Can you intercept the job anywhere? Stephan On Mon, Aug 29, 2016 at 8:47 AM, Bhupesh Chawda

[jira] [Created] (FLINK-4524) JobManagerProcessReapingTest.testReapProcessOnFailure unstable

Robert Metzger created FLINK-4524: - Summary: JobManagerProcessReapingTest.testReapProcessOnFailure unstable Key: FLINK-4524 URL: https://issues.apache.org/jira/browse/FLINK-4524 Project: Flink

Re: Enabling Encryption between slaves in Flink

Hi Vinay, it's not only done by enabling encryption of the Akka messages since the TaskManager to TaskManager communication utilizes Netty. There is a JIRA issue which tracks the progress on implementing secured communication within Flink: https://issues.apache.org/jira/browse/FLINK-3931. Feel

Yarn node went down...

Hi, Last week I brought down one of our Yarn nodes because of this problem: https://issues.apache.org/jira/browse/FLINK-4485 The Yarn node no longer accepted any Flink/Yarn jobs until we stopped the jobmaster. I classified this as a 'blocker' because it makes the yarn-session very short lived

Check-pointing configuration

Hi, I have a question on enabling checkpointing for one of my Flink jobs. I understand that I can call *enableCheckpointing* on the *StreamExecutionEnvironment* to enable it. In my case, I don't have access to the source code; Is there also an option to enable it through configuration? Thanks.