[jira] [Created] (FLINK-3691) Extend AvroInputFormat to support Avro GenericRecord

[Gna Phetsarath (JIRA)]

Gna Phetsarath created FLINK-3691:
-

 Summary: Extend AvroInputFormat to support Avro GenericRecord
 Key: FLINK-3691
 URL: https://issues.apache.org/jira/browse/FLINK-3691
 Project: Flink
  Issue Type: Improvement
Reporter: Gna Phetsarath


Extend AvroInputFormat to support Avro GenericRecord: 

Sometimes the calling application will have access to the schema but not the 
implementing class for Avro files.  

Using a GenericDatumReader will allow the AvroInput format to read records in 
the Avro Generic Record Format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Failing Test

[Maximilian Michels]

Fixed with the resolution of https://issues.apache.org/jira/browse/FLINK-3689.

On Fri, Apr 1, 2016 at 12:40 PM, Maximilian Michels  wrote:
> Hi Matthias,
>
> Thanks for spotting the test failure. It's actually a bug in the code
> and not a test problem. Fixing it.
>
> Cheers,
> Max
>
> On Fri, Apr 1, 2016 at 9:33 AM, Ufuk Celebi  wrote:
>> Hey Matthias,
>>
>> the test has been only recently added with the resource management
>> refactoring. It's probably just a too aggressive timeout for Travis.
>>
>> @Max: Did you ever see this fail?
>>
>> – Ufuk
>>
>> On Fri, Apr 1, 2016 at 9:24 AM, Matthias J. Sax  wrote:
>>> Anyone seen this before? One-time thing or test instability?
>>>
 ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout 
 (29848225634 nanoseconds) during expectMsgClass waiting for class 
 org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful
>>>
>>>
>>> -Matthias
>>>

[jira] [Created] (FLINK-3690) Create documentation on the new ResourceManager component

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-3690:


 Summary: Create documentation on the new ResourceManager component
 Key: FLINK-3690
 URL: https://issues.apache.org/jira/browse/FLINK-3690
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, ResourceManager
Affects Versions: 1.1.0
Reporter: Henry Saputra
Assignee: Maximilian Michels


Need proper documentation for the new ResourceManager and how it will impact 
deployment in different supported modes.

Also, we have been very good adding new docs for our internal in the wiki [1] 
so would like that to happen for people evaluating Flink.

[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Announcing S-link-y

[Aljoscha Krettek]

perfect

On Fri, 1 Apr 2016 at 15:51 Bobby Evans  wrote:

> Today I am happy to announce a prototype of an Apache Storm based backend
> for Apache Flink called S-link-y.  Storm offers a number of advantages over
> Flink including enterprise level security and a distributed cache with in
> place updating.  The Flink front end offers both batch processing APIs and
> an Apache BEAM implementation.   As part of our testing we were surprised
> to discover that running the Storm WordCountTopology through the
> Flink-Storm compatibility layer on the S-link-y backend had 89% latency and
> 3x the throughput of running it on Storm itself and 78% the latency of a
> native Flink WordCount.
> To see the benchmark results, download the source code and try it out
> yourself please go to http://bit.ly/1gho7il
> We would love to begin discussions between the Flink and Storm PMCs about
> merging the two projects.
>
> - Bobby

CEP blog post

[Till Rohrmann]

Hi Flink community,

I've written a short blog [1] post about Flink's new CEP library which
basically showcases its functionality using a monitoring example. I would
like to publish the post on the flink.apache.org blog next week, if nobody
objects. Feedback is highly appreciated :-)

[1]
https://docs.google.com/document/d/1rF2zVjitdTcooIwzJKNCIvAOi85j-wDXf1goXWXHHbk/edit?usp=sharing

Cheers,
Till

Announcing S-link-y

[Bobby Evans]

Today I am happy to announce a prototype of an Apache Storm based backend for 
Apache Flink called S-link-y.  Storm offers a number of advantages over Flink 
including enterprise level security and a distributed cache with in place 
updating.  The Flink front end offers both batch processing APIs and an Apache 
BEAM implementation.   As part of our testing we were surprised to discover 
that running the Storm WordCountTopology through the Flink-Storm compatibility 
layer on the S-link-y backend had 89% latency and 3x the throughput of running 
it on Storm itself and 78% the latency of a native Flink WordCount.
To see the benchmark results, download the source code and try it out yourself 
please go to http://bit.ly/1gho7il
We would love to begin discussions between the Flink and Storm PMCs about 
merging the two projects.

- Bobby

Re: [VOTE] Release Apache Flink 1.0.1 (RC1)

[Aljoscha Krettek]

I ran the usual tests and the release seems fine. A user, however, found
this bug which seems to be a blocker:
https://issues.apache.org/jira/browse/FLINK-3688?jql=project%20%3D%20FLINK

Should we maybe wait for the fix and then to another RC?

On Thu, 31 Mar 2016 at 18:10 Ufuk Celebi  wrote:

> Dear Flink community,
>
> Please vote on releasing the following candidate as Apache Flink version
> 1.0.1.
>
> The commit to be voted on:
> 4afa401ab3c2b53de115d17a3157e8b80431dd10
>
> Branch:
> release-1.0.1-rc1 (see
>
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-1.0.1-rc1
> )
>
> The release artifacts to be voted on can be found at:
> http://home.apache.org/~uce/flink-1.0.1-rc1/
>
> The release artifacts are signed with the key with fingerprint 9D403309:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapacheflink-1089
>
> -
>
> The vote is open for the next 72 hours and passes if a majority of at least
> three +1 PMC votes are cast.
>
> The vote ends on Tuesday April 5, 2016 (not counting the weekend).
>
> [ ] +1 Release this package as Apache Flink 1.0.1
> [ ] -1 Do not release this package because ...
>
> ===
>
> The following commits have been added since the 1.0.0 release, most
> notably a fix for the rolling HDFS file sink and support for
> RichFunctions in Scala.
>
> * 974c66e - [FLINK-3383] Backport 'Move snapshot deployment from
> Travis CI to ASF Jenkins' (6 hours ago) 
> * 630e77f - [FLINK-3684] [cep] Add proper watermark emission to CEP
> operators (6 hours ago) 
> * dae29b4 - [FLINK-3682] [cep] Assign processing timestamp in CEP
> operators (23 hours ago) 
> * a96e1a6 - [FLINK-3681] [cep, typeextractor] Generalize TypeExtractor
> to support more lambdas (23 hours ago) 
> * ce448cd - [tools] Update copy files function in release script (25
> hours ago) 
> * 245f0d8 - [hotfix] [cep] Add sanity check for erroneously pruned
> elements (26 hours ago) 
> * 2089029 - [FLINK-3651] Fix faulty RollingSink Restore (2 days ago)
> 
> * 875cb44 - [FLINK-3676][docs] sync docs with the WebClient removal (2
> days ago) 
> * 085f7b2 - [FLINK-3663] [kafka] Fix logPartitionInfo log statement (7
> days ago) 
> * 5357ebe - [FLINK-3653] [docs] Add recovery.zookeeper.storageDir docs
> to config page (7 days ago) 
> * 1ca4f36 - [FLINK-2444] Add tests for HadoopInputFormats (8 days ago)
> 
> * da989c4 - [FLINK-2445] Improve HadoopOutputFormatTests (8 days ago)
> 
> * edf343a - [FLINK-2445] Add tests for HadoopOutputFormats (8 days
> ago) 
> * 1909bdb - [FLINK-3129] Bring infrastructure for ensuring interface
> stability in place (8 days ago) 
> * cd5773e - [FLINK-3179] [dataSet][optimizer] Log a WARN message if
> combiner is not added in front of PartitionOperator (9 days ago)
> 
> * 0880c59 - [FLINK-3602] Fix TypeExtractor and add support for
> recursive types (9 days ago) 
> * 954cdc1 - [hotfix] Fix ContinuousProcessingTimeTrigger (10 days ago)
> 
> * 2c78be3 - [FLINK-3629] Fix quick start description for the
> ".timeWindow()" function call (2 weeks ago) 
> * 6749874 - [FLINK-3621] [docs] Improve documentation of memory
> configuration parameters (2 weeks ago) 
> * 165065c - [FLINK-] [docs] Improve documentation of DataSet
> object-reuse modes. (2 weeks ago) 
> * 3d8c7c1 - [FLINK-3472] [jdbc] Give a better exception if jdbc column
> has a null value (3 weeks ago) 
> * 43e95f2 - [FLINK-2671] [tests] Fix test instability in
> StreamCheckpointNotifierITCase (3 weeks ago) 
> * f905503 - [FLINK-3594] [runtime] Make sure exceptions during
> checkpoints are handled properly (3 weeks ago) 
> * a405b55 - [hotfix] [runtime] Guard async recovery operation in
> try-catch (3 weeks ago) 
> * 5f35d13 - [quickstart] fix links to documentation (3 weeks ago)
> 
> * dc675f2 - [docs] Remove 'Working with Time' page (3 weeks ago)  Celebi>
> * 95cd07c - [docs] Add description and illustration about Watermarks
> in streams (3 weeks ago) 
> * 2831b7d - [docs] Add first steps to docs start page (3 weeks ago)
> 
> * a709f0a - [docs] Update stack figure on docs start page (3 weeks
> ago) 
> * bb9c1fa - [FLINK-3592] [docs] Update setup quickstart (3 weeks ago)
> 
> * d4c2a47 - [FLINK-3591] Replace Quickstart K-Means Example by
> Streaming Example (3 weeks ago) 
> * 2f3711b - [docs] Move connectors page to batch guide (3 weeks ago)
> 
> * db860e4 - Update version and remove unused variables (3 weeks ago)
> 
> * 4f6a92d - [docs] update "build from source" page (3 weeks ago)
> 
> * a08242f - [FLINK-3533] [docs] fix broken links to examples and
> cluster execution in Gelly guide (4 weeks ago) 
> * b376540 - [FLINK-3566] [FLINK-3563] [core] TypeExtraction input type
> validation fixes (4 weeks ago) 
> * 2cde19b - [FLINK-3495] [tests] Disable RocksDB tests on Windows (4
> weeks ago) 
> * 

Re: Failing Test

[Maximilian Michels]

Hi Matthias,

Thanks for spotting the test failure. It's actually a bug in the code
and not a test problem. Fixing it.

Cheers,
Max

On Fri, Apr 1, 2016 at 9:33 AM, Ufuk Celebi  wrote:
> Hey Matthias,
>
> the test has been only recently added with the resource management
> refactoring. It's probably just a too aggressive timeout for Travis.
>
> @Max: Did you ever see this fail?
>
> – Ufuk
>
> On Fri, Apr 1, 2016 at 9:24 AM, Matthias J. Sax  wrote:
>> Anyone seen this before? One-time thing or test instability?
>>
>>> ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout 
>>> (29848225634 nanoseconds) during expectMsgClass waiting for class 
>>> org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful
>>
>>
>> -Matthias
>>

[jira] [Created] (FLINK-3688) ClassCastException in StreamRecordSerializer when WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime

[Konstantin Knauf (JIRA)]

Konstantin Knauf created FLINK-3688:
---

 Summary: ClassCastException in StreamRecordSerializer when 
WindowOperator.trigger() is called and TimeCharacteristic = ProcessingTime
 Key: FLINK-3688
 URL: https://issues.apache.org/jira/browse/FLINK-3688
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Konstantin Knauf
Priority: Critical


Hi,

when using {{TimeCharacteristics.ProcessingTime}} a ClassCastException is 
thrown in {{StreamRecordSerializer}} when {{WindowOperator.processWatermark()}} 
is called from {{WindowOperator.trigger()}}, i.e. whenever a 
ProcessingTimeTimer is triggered. 

The problem seems to be that {{processWatermark()}} is also called in 
{{trigger()}}, when time characteristic is ProcessingTime, but in 
{{RecordWriterOutput}} {{enableWatermarkMultiplexing}} is {{false}} and the 
{{TypeSerializer}} is a {{StreamRecordSerializer}}, which ultimately leads to 
the ClassCastException. Do you agree?

If this is indeed a bug, there several possible solutions.

# Only calling {{processWatermark()}} in {{trigger()}}, when TimeCharacteristic 
is EventTime
# Not calling {{processWatermark()}} in {{trigger()}} at all, instead wait for 
the next watermark to trigger the EventTimeTimers with a timestamp behind the 
current watermark. This is, of course, a trade off. 
# Using {{MultiplexingStreamRecordSerializer}} all the time, but I have no idea 
what the side effect of this change would be. I assume there is a reason for 
existence of the StreamRecordSerializer ;)

StackTrace: 
{quote}
TimerException\{java.lang.RuntimeException: 
org.apache.flink.streaming.api.watermark.Watermark cannot be cast to 
org.apache.flink.streaming.runtime.streamrecord.StreamRecord\}
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:716)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: 
org.apache.flink.streaming.api.watermark.Watermark cannot be cast to 
org.apache.flink.streaming.runtime.streamrecord.StreamRecord
at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitWatermark(OperatorChain.java:370)
at 
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processWatermark(WindowOperator.java:293)
at 
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.trigger(WindowOperator.java:323)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$TriggerTask.run(StreamTask.java:710)
... 7 more
Caused by: java.lang.ClassCastException: 
org.apache.flink.streaming.api.watermark.Watermark cannot be cast to 
org.apache.flink.streaming.runtime.streamrecord.StreamRecord
at 
org.apache.flink.streaming.runtime.streamrecord.StreamRecordSerializer.serialize(StreamRecordSerializer.java:42)
at 
org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
at 
org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
at 
org.apache.flink.runtime.io.network.api.writer.RecordWriter.broadcastEmit(RecordWriter.java:109)
at 
org.apache.flink.streaming.runtime.io.StreamRecordWriter.broadcastEmit(StreamRecordWriter.java:95)
at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:90)
... 11 more
{quote}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3687) runtime.leaderelection.ZooKeeperLeaderElectionTest fails

[Nikolaas Steenbergen (JIRA)]

Nikolaas Steenbergen created FLINK-3687:
---

 Summary: runtime.leaderelection.ZooKeeperLeaderElectionTest fails
 Key: FLINK-3687
 URL: https://issues.apache.org/jira/browse/FLINK-3687
 Project: Flink
  Issue Type: Bug
Reporter: Nikolaas Steenbergen


Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.51 sec <<< 
FAILURE! - in org.apache.flink.runtime.net.ConnectionUtilsTest
testReturnLocalHostAddressUsingHeuristics(org.apache.flink.runtime.net.ConnectionUtilsTest)
  Time elapsed: 0.504 sec  <<< FAILURE!
java.lang.AssertionError: 
expected: but 
was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.flink.runtime.net.ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics(ConnectionUtilsTest.java:59)

{code}
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.554 sec - in 
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest

Results :

Failed tests: 
  ConnectionUtilsTest.testReturnLocalHostAddressUsingHeuristics:59 
expected: but 
was:
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3686) resourcemanager.ClusterShutdownITCase sometimes fails

[Nikolaas Steenbergen (JIRA)]

Nikolaas Steenbergen created FLINK-3686:
---

 Summary: resourcemanager.ClusterShutdownITCase sometimes fails
 Key: FLINK-3686
 URL: https://issues.apache.org/jira/browse/FLINK-3686
 Project: Flink
  Issue Type: Bug
  Components: Tests
Reporter: Nikolaas Steenbergen


resourcemanager.ClusterShutdownITCase sometimes fails

```Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.47 sec 
<<< FAILURE! - in org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase
testClusterShutdown(org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase)
  Time elapsed: 30.063 sec  <<< FAILURE!
java.lang.AssertionError: assertion failed: timeout (29897828577 nanoseconds) 
during expectMsgClass waiting for class 
org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful
at scala.Predef$.assert(Predef.scala:179)
at 
akka.testkit.TestKitBase$class.expectMsgClass_internal(TestKit.scala:423)
at akka.testkit.TestKitBase$class.expectMsgClass(TestKit.scala:410)
at akka.testkit.TestKit.expectMsgClass(TestKit.scala:718)
at akka.testkit.JavaTestKit.expectMsgClass(JavaTestKit.java:397)
at 
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.run(ClusterShutdownITCase.java:92)
at akka.testkit.JavaTestKit$Within$1.apply(JavaTestKit.java:232)
at akka.testkit.TestKitBase$class.within(TestKit.scala:296)
at akka.testkit.TestKit.within(TestKit.scala:718)
at akka.testkit.TestKitBase$class.within(TestKit.scala:310)
at akka.testkit.TestKit.within(TestKit.scala:718)
at akka.testkit.JavaTestKit$Within.(JavaTestKit.java:230)
at 
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1$1.(ClusterShutdownITCase.java:72)
at 
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase$1.(ClusterShutdownITCase.java:72)
at 
org.apache.flink.runtime.resourcemanager.ClusterShutdownITCase.testClusterShutdown(ClusterShutdownITCase.java:71)


Results :

Failed tests: 
  ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout 
(29897828577 nanoseconds) during expectMsgClass waiting for class 
org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Failing Test

[Ufuk Celebi]

Hey Matthias,

the test has been only recently added with the resource management
refactoring. It's probably just a too aggressive timeout for Travis.

@Max: Did you ever see this fail?

– Ufuk

On Fri, Apr 1, 2016 at 9:24 AM, Matthias J. Sax  wrote:
> Anyone seen this before? One-time thing or test instability?
>
>> ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout 
>> (29848225634 nanoseconds) during expectMsgClass waiting for class 
>> org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful
>
>
> -Matthias
>

Failing Test

[Matthias J. Sax]

Anyone seen this before? One-time thing or test instability?

> ClusterShutdownITCase.testClusterShutdown:71 assertion failed: timeout 
> (29848225634 nanoseconds) during expectMsgClass waiting for class 
> org.apache.flink.runtime.clusterframework.messages.StopClusterSuccessful


-Matthias



signature.asc
Description: OpenPGP digital signature