[jira] [Created] (FLINK-8430) Implement stream-stream non-window full outer join

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-8430:
--

 Summary: Implement stream-stream non-window full outer join
 Key: FLINK-8430
 URL: https://issues.apache.org/jira/browse/FLINK-8430
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Reporter: Hequn Cheng
Assignee: Hequn Cheng






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8429) Implement stream-stream non-window right outer join

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-8429:
--

 Summary: Implement stream-stream non-window right outer join
 Key: FLINK-8429
 URL: https://issues.apache.org/jira/browse/FLINK-8429
 Project: Flink
  Issue Type: Sub-task
Reporter: Hequn Cheng
Assignee: Hequn Cheng






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8428) Implement stream-stream non-window left outer join

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-8428:
--

 Summary: Implement stream-stream non-window left outer join
 Key: FLINK-8428
 URL: https://issues.apache.org/jira/browse/FLINK-8428
 Project: Flink
  Issue Type: Sub-task
Reporter: Hequn Cheng
Assignee: Hequn Cheng






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8427) Checkstyle for org.apache.flink.optimizer.costs

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-8427:
-

 Summary: Checkstyle for org.apache.flink.optimizer.costs
 Key: FLINK-8427
 URL: https://issues.apache.org/jira/browse/FLINK-8427
 Project: Flink
  Issue Type: Improvement
  Components: Optimizer
Affects Versions: 1.5.0
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8426) Error in Generating Timestamp/Watermakr doc

[Christophe Jolif (JIRA)]

Christophe Jolif created FLINK-8426:
---

 Summary: Error in Generating Timestamp/Watermakr doc
 Key: FLINK-8426
 URL: https://issues.apache.org/jira/browse/FLINK-8426
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.4.0
Reporter: Christophe Jolif
Priority: Trivial


In 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html

{{
public class BoundedOutOfOrdernessGenerator extends 
AssignerWithPeriodicWatermarks}}

should be
{{
public class BoundedOutOfOrdernessGenerator implements 
AssignerWithPeriodicWatermarks}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8425) SpilledSubpartitionView not protected against concurrent release calls

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-8425:
--

 Summary: SpilledSubpartitionView not protected against concurrent 
release calls
 Key: FLINK-8425
 URL: https://issues.apache.org/jira/browse/FLINK-8425
 Project: Flink
  Issue Type: Bug
  Components: Network
Affects Versions: 1.3.2, 1.4.0, 1.3.1, 1.2.1, 1.3.0, 1.1.4, 1.1.3, 1.1.2, 
1.1.1, 1.2.0, 1.0.3, 1.0.2, 1.0.1, 1.1.0
Reporter: Nico Kruber
Priority: Minor


It seems like {{SpilledSubpartitionView}} is not protected against concurrently 
calling {{releaseAllResources}} as the other {{ResultSubpartitionView}} 
implementations. These may happen due to failures, e.g. network channels 
breaking, and will probably only result in some unexpected exceptions being 
thrown, e.g. from reading from a closed file reader.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

difference b/t filesystem and connector-filesystem

[cw7k]

Hi, I'm trying to understand the difference between the flink-filesystem and 
flink-connector-filesystem.  How is each intended to be used?
If adding support for a different storage provider that supports HDFS, should 
additions be made to one or the other, or both?  Thanks.

[jira] [Created] (FLINK-8424) [Cassandra Connector] Update Cassandra version to last one

[Joao Boto (JIRA)]

Joao Boto created FLINK-8424:


 Summary: [Cassandra Connector] Update Cassandra version to last one
 Key: FLINK-8424
 URL: https://issues.apache.org/jira/browse/FLINK-8424
 Project: Flink
  Issue Type: Improvement
Reporter: Joao Boto
Priority: Critical


Cassandra connector are using a version release in the beginning of 2016

This is to upgrade the cassandra version to something new




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8423) OperatorChain#pushToOperator catch block may fail with NPE

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-8423:
---

 Summary: OperatorChain#pushToOperator catch block may fail with NPE
 Key: FLINK-8423
 URL: https://issues.apache.org/jira/browse/FLINK-8423
 Project: Flink
  Issue Type: Bug
  Components: Streaming
Affects Versions: 1.4.0, 1.5.0
Reporter: Chesnay Schepler
Priority: Minor


{code}
@Override
protected  void pushToOperator(StreamRecord record) {
try {
// we know that the given outputTag matches our OutputTag so 
the record
// must be of the type that our operator (and Serializer) 
expects.
@SuppressWarnings("unchecked")
StreamRecord castRecord = (StreamRecord) record;

numRecordsIn.inc();
StreamRecord copy = 
castRecord.copy(serializer.copy(castRecord.getValue()));
operator.setKeyContextElement1(copy);
operator.processElement(copy);
} catch (ClassCastException e) {
// Enrich error message
ClassCastException replace = new ClassCastException(
String.format(
"%s. Failed to push OutputTag with id '%s' to 
operator. " +
"This can occur when multiple OutputTags with 
different types " +
"but identical names are being used.",
e.getMessage(),
outputTag.getId()));

throw new ExceptionInChainedOperatorException(replace);

} catch (Exception e) {
throw new ExceptionInChainedOperatorException(e);
}
}
{code}

If outputTag is null (as is the case when no sideOutput was defined) the catch 
block will crash with a NullPointerException. This may happen if 
{{operator.processElement}} throws a ClassCastException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8422) Checkstyle for org.apache.flink.api.java.tuple

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-8422:
-

 Summary: Checkstyle for org.apache.flink.api.java.tuple
 Key: FLINK-8422
 URL: https://issues.apache.org/jira/browse/FLINK-8422
 Project: Flink
  Issue Type: Improvement
  Components: Core
Affects Versions: 1.5.0
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Trivial


Update {{TupleGenerator}} for Flink's checkstyle and rebuild {{Tuple}} and 
{{TupleBuilder}} classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8421) HeapInternalTimerService should reconfigure compatible key / namespace serializers on restore

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-8421:
--

 Summary: HeapInternalTimerService should reconfigure compatible 
key / namespace serializers on restore
 Key: FLINK-8421
 URL: https://issues.apache.org/jira/browse/FLINK-8421
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.4.0, 1.5.0
Reporter: Tzu-Li (Gordon) Tai
Priority: Blocker
 Fix For: 1.3.3, 1.5.0, 1.4.1


The {{HeapInternalTimerService}} still uses simple {{equals}} checks on 
restored / newly provided serializers for compatibility checks. This should be 
replaced with the {{TypeSerializer::ensureCompatibility}} checks instead, so 
that new serializers can be reconfigured.

For Flink 1.4.0 release and current master, this is a critical bug since the 
{{KryoSerializer}} has different default base registrations than before due to 
FLINK-7420. i.e if the key of a window is serialized using the 
{{KryoSerializer}} in 1.3.x, the restore would never succeed in 1.4.0.

For 1.3.x, this fix would be an improvement, such that the 
{{HeapInternalTimerService}} restore will make use of serializer 
reconfiguration.

Other remarks:
* We need to double check all operators that checkpoint / restore from **raw** 
state. Apparently, the serializer compatibility checks were only implemented 
for managed state.
* Migration ITCases apparently do not have enough coverage. A migration test 
job that uses a key type which required the {{KryoSerializer}}, and uses 
windows, would have caught this issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8420) Timeout exceptions are not properly recognized by RetryingRegistration

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-8420:


 Summary: Timeout exceptions are not properly recognized by 
RetryingRegistration
 Key: FLINK-8420
 URL: https://issues.apache.org/jira/browse/FLINK-8420
 Project: Flink
  Issue Type: Bug
  Components: Distributed Coordination
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.5.0


The {{RetryingRegistration}} does not correctly respond to 
{{TimeoutExceptions}} and instead treats them like errors. This causes that it 
waits for the delay on error instead of backing exponentially off.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Ordering in ProcessFunction after emitting multiple events from WindowFunction

[Michał Stępień]

Hello,

I have dev related question.

How will Flink behave when processing events in WindowFunciton (version 1.3
or 1.4 including ProcessWindowFuncton) produces multiple events? Does Flink
guarantee that ProcessFunction will process them in order after window
outputs them in order into Collector for given key? Moreover, I understand
that Flink won't process next event in ProcessFunction as long as it won't
finish previous one for given key.

Code:

env.addSource(eventSource)
.assignTimestampsAndWatermarks(timestampsAndWatermarksGenerator)
.keyBy(FLINK_GROUPING_FIELD_NAME)

.window(TumblingEventTimeWindows.of(Time.milliseconds((WINDOW_DURATION_MILLIS
.apply(windowFunction)
.keyBy(FLINK_GROUPING_FIELD_NAME)
.process(processFunction)
.addSink(sinkFunction);

-- 
Pozdrawiam/Regards,
Michał

*Michał Stępień*
Developer
*freeportmetrics.com *  |  +48 22 253 25 13
<%2B48%20660%20507%20111>

Re: Dynamic SessionWindow gaps

[dyana.r...@salecycle.com]

I've put up some implementation information and a link to the working branch 
for comment while I get the documentation written up: 

https://issues.apache.org/jira/browse/FLINK-8384

Thanks,
Dyana

On 2018-01-02 13:54, jincheng sun  wrote: 
> Hi, Dyana dynamic gap is very useful feature, Agree with the suggestion of
> Aljoscha, looking forward the PR.
> 
> Best, Jincheng
> 
> 2018-01-02 21:45 GMT+08:00 Aljoscha Krettek :
> 
> > Hi,
> >
> > This sounds good! Could you please open a Jira Issue for this and then
> > open a PR against that Jira issue?
> >
> > Btw, I would suggest to implement this as a new type of assigner,
> > something like DynamicSessionWindows.
> >
> > Best,
> > Aljoscha
> >
> > > On 29. Dec 2017, at 20:54, Dyana Rose  wrote:
> > >
> > > I have a use case for non-static Session Window gaps.
> > >
> > > For example, given a stream of IoT events, each device type could have a
> > > different gap, and that gap could change while sessions are in flight.
> > >
> > > I didn't want to have to run a stream processor for each potential gap
> > > length, not to mention the headache of dealing with changing gaps, so
> > I've
> > > implemented a version of SessionWindows that has one major change; in the
> > > assignWindows method it passes the element to a method to extract the
> > > correct sessionTimeout. (current Flink method for reference:
> > > https://github.com/apache/flink/blob/master/flink-
> > streaming-java/src/main/java/org/apache/flink/streaming/
> > api/windowing/assigners/EventTimeSessionWindows.java#L59
> > > )
> > >
> > > Preliminary tests show this working as required and I can't be the only
> > > person with this type of use case for session windows.
> > >
> > > Will an issue and PR to add this functionality to the SessionWindow
> > classes
> > > be welcome?
> > >
> > > Dyana
> >
> >
>