[jira] [Created] (FLINK-8399) Use independent configurations for the different timeouts in slot manager

[shuai.xu (JIRA)]

shuai.xu created FLINK-8399:
---

 Summary: Use independent configurations for the different timeouts 
in slot manager
 Key: FLINK-8399
 URL: https://issues.apache.org/jira/browse/FLINK-8399
 Project: Flink
  Issue Type: Bug
  Components: Cluster Management
Affects Versions: 1.5.0
Reporter: shuai.xu
Assignee: shuai.xu


There are three parameter in slot manager to indicate the timeout for slot 
request to task manager, slot request to be discarded and task manager to be 
released. But now the all come from the value of AkkaOptions.ASK_TIMEOUT, need 
to use independent configurations for them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8398) Stabilize flaky KinesisDataFetcherTests

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-8398:
--

 Summary: Stabilize flaky KinesisDataFetcherTests
 Key: FLINK-8398
 URL: https://issues.apache.org/jira/browse/FLINK-8398
 Project: Flink
  Issue Type: Bug
  Components: Kinesis Connector, Tests
Affects Versions: 1.4.0, 1.5.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
 Fix For: 1.5.0, 1.4.1


The unit tests in {{KinesisDataFetcherTest}} have very flaky implementations. 
They rely on on thread sleeps to wait for a certain operation to happen, which 
can easily miss and cause tests to fail.

Although there isn't any reports of consistent failures on these tests yet (as 
far as I am aware of),  they can easily surface in the future.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8397) Support ROW type in CassandraOutputFormat

[Shuyi Chen (JIRA)]

Shuyi Chen created FLINK-8397:
-

 Summary: Support ROW type in CassandraOutputFormat
 Key: FLINK-8397
 URL: https://issues.apache.org/jira/browse/FLINK-8397
 Project: Flink
  Issue Type: Improvement
 Environment: Currently, only tuple is supported.
Reporter: Shuyi Chen
Assignee: Shuyi Chen






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8396) Create (derived) duplicate Buffer class

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-8396:
--

 Summary: Create (derived) duplicate Buffer class
 Key: FLINK-8396
 URL: https://issues.apache.org/jira/browse/FLINK-8396
 Project: Flink
  Issue Type: Sub-task
  Components: Network
Reporter: Nico Kruber
Assignee: Nico Kruber


In order to pass a single buffer to netty multiple times, we require a 
duplicate Buffer instance with a shared {{MemorySegment}} and reference 
counting but separate indices.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8395) Create (derived) sliced Buffer class

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-8395:
--

 Summary: Create (derived) sliced Buffer class
 Key: FLINK-8395
 URL: https://issues.apache.org/jira/browse/FLINK-8395
 Project: Flink
  Issue Type: Sub-task
  Components: Network
Reporter: Nico Kruber
Assignee: Nico Kruber


In order to pass sub-regions of a single buffer separately, we require a sliced 
Buffer instance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8394) Lack of synchronization accessing expectedRecord in ReceiverThread#shutdown

[Ted Yu (JIRA)]

Ted Yu created FLINK-8394:
-

 Summary: Lack of synchronization accessing expectedRecord in 
ReceiverThread#shutdown
 Key: FLINK-8394
 URL: https://issues.apache.org/jira/browse/FLINK-8394
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  public void shutdown() {
running = false;
interrupt();
expectedRecord.complete(0L);
{code}
Access to expectedRecord should be protected by synchronization, as done on 
other methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8393) Reconnect to last known JobMaster when connection is lost

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-8393:


 Summary: Reconnect to last known JobMaster when connection is lost
 Key: FLINK-8393
 URL: https://issues.apache.org/jira/browse/FLINK-8393
 Project: Flink
  Issue Type: Bug
  Components: Distributed Coordination
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.5.0


In case of a connection loss to the {{JobMaster}}, e.g. due to a heartbeat 
timeout or a disconnect call, then the {{TaskExecutor}} should try to reconnect 
to the last known {{JobMaster}} location in case that the timeout was a false 
positive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: aggregate stream of logs

[Gary Yao]

Hi,

First of all, your question is more suited for the Flink user mailing list
(this
is the dev mailing list).

I think you can use a GlobalWindow [1] with a custom trigger. A global
window
assigns all elements with the same key to the same window, and will not
perform
any computation unless triggered. You may want to look into the "Connected
Car
Segments" Flink training exercise and its reference solution, which I think
solves a similar problem:

  http://training.data-artisans.com/exercises/carSegments.html

Best,
Gary

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/windows.html#global-windows

On Mon, Jan 8, 2018 at 4:40 PM, miki haiat  wrote:

> Hi ,
>
> i have this scenario of applications that streaming logs to kafka .
>
> I want to use flink in order to aggregate the stream and calculate some
> data
> i have a guid that i can correlate the stream to single event   and i have
> a field  that i can use to understand if is the last stream of the event .
>
> lets say the stream is something like that ,
>
> 92e8366c-8ddd-41ed-a297-f9c28bf38792msg1start
> 92e8366c-8ddd-41ed-a297-f9c28bf38792msg2start
> 92e8366c-8ddd-41ed-a297-f9c28bf38792msg3end
> 92e8366c-8ddd-41ed-a297-f9c28bf3666  msg1start
>
> I  cant use window trigger  because then i can lose some data ?
>
> the only thing i can think about is onElement trigger ?
>
> what is you suggestion for this use case ?
>

[jira] [Created] (FLINK-8392) Simplify termination future completion

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-8392:


 Summary: Simplify termination future completion
 Key: FLINK-8392
 URL: https://issues.apache.org/jira/browse/FLINK-8392
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Coordination
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann


With FLINK-7754, we tried to complete the termination future after an {{Actor}} 
has been completely stopped and has been removed from the {{ActorSystem}}. 
This, however, is not possible. Furthermore, this change made it impossible 
that a {{RpcEndpoint}} waits for the termination of another {{RpcEndpoint}} in 
its {{RpcEndpoint#postStop}} method. Therefore, I propose to revert the changes 
done by FLINK-7754.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Issues to work on for newbie to the Flink project

[Ashutosh Dubey]

Hello Flink Commiters,

I have been reading Flink documentation for some time now, and want to
contribute as a developer to the project, however unlike other projects I
could not find tags on issues that suggest it suitable for newbies to pick
up.

I would really appreciate if you could suggest few jira issues that I could
pick up from to get started with.

I appreciate your time and efforts for the community.

Thanks!

Ashutosh

Re: [DISCUSS] FLIP-24 - SQL Client

[Timo Walther]

Hi Fabian,

thank you for your feedback. I extended the "Implementation Plan" by 3 
subsections. Regarding the proposed changes, I looked into the code of 
the ClusterClient and talked to Till Rohrmann about it. By now, I could 
not identify features that are missing on existing components. It might 
be necessary to make some internal methods accessible but no big 
changes. I also extended the "Result Retrieval" section to (hopefully) 
answer your questions.


Feel free to give further comments.

Regards,
Timo



Am 12/21/17 um 3:52 PM schrieb Fabian Hueske:

Hi Timo,

thanks for putting this FLIP together. I think this will be a great feature
for Flink.

I think it makes sense to show the long term goals of the SQL client/server
architecture, but the current description makes it a bit difficult to
figure out what will be part of this FLIP and what will be added later
because the information is spread across the FLIP. This could be improved
by describing the "Implementation Plan" in more detail, i.e, have
subsections for the first version with a detailed list of supported
features and another for later extensions.

Comments to specific sections:
- Proposed changes: Are there any changes necessary on existing code (e.g.,
job submission with multiple jars) or are all changes just additions to
Flink?
- Result Retrieval: Extend the table to three columns (Query type, Internal
mode, External mode).
- The context of the description of the query types "CREATE MATERIALIZED
VIEW" and "SELECT" is not clear to me. Add a subsection and explain what is
described there?
- Implementation plan: Add which result retrieval modes will be supported
in the initial version? Which configuration will be available?

Best, Fabian

2017-12-19 16:53 GMT+01:00 Timo Walther :


Hey everyone,

in the past, the community already asked about having a way to write Flink
jobs without extensive programming skills. During the last year we have put
a lot of effort in the core of our Table & SQL API. Now it is time to
improve the tooling around it as well and make Flink more accessible. I
already opened an issue for adding a SQL CLI client [0]. We developed a
small protoype at data Artisans that was shown at the last Flink Forward
Berlin [1].

For Flink 1.5 it would be great to offer at least a CLI client to play
around with Flink and use it for debugging/demo purposes. We created a
FLIP-24 [2] that roughly sketches the functionality and architecture. We
also show how this architecture can evolve from a CLI client to a SQL
gateway/REST server. Most of it is still up for dicussion. The targeted
minimum viable product for Flink 1.5 should act as a start for collecting
feedback and attracting contributors.

So feedback on this FLIP is very welcome. What do you think?

Regards,
Timo


[0] https://issues.apache.org/jira/browse/FLINK-7594
[1] https://berlin-2017.flink-forward.org/kb_sessions/from-strea
ms-to-tables-and-back-again-a-demo-of-flinks-table-sql-api/
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client

[jira] [Created] (FLINK-8391) Improve dependency documentation for queryable state

[Fabian Hueske (JIRA)]

Fabian Hueske created FLINK-8391:


 Summary: Improve dependency documentation for queryable state
 Key: FLINK-8391
 URL: https://issues.apache.org/jira/browse/FLINK-8391
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.5.0, 1.4.1
Reporter: Fabian Hueske


The documentation of the queryable state does not explicitly mention that the 
program that holds and updates the queryable state requires the 
{{flink-queryable-state-runtime}} dependency.

Right now, the section "Activating Queryable State" mentions, that 
{{flink-queryable-state-runtime}} needs to be copied into the {{./lib}} folder. 
However, this does not work if the program is started from an IDE, for example 
for local testing.

The docs should be updated accordingly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)