Performance and Latency Chart for Flink

2016-09-15 Thread Chawla,Sumit 
Hi

Is there any performance run that is done for each Flink release? Or you
are aware of any third party evaluation of performance metrics for Flink?
I am interested in seeing how performance has improved over release to
release, and performance vs other competitors.

Regards
Sumit Chawla

[jira] [Created] (FLINK-4627) Use Flink's PropertiesUtil in Kinesis connector to extract typed values from config properties

2016-09-15 Thread Tzu-Li (Gordon) Tai (JIRA) 
Tzu-Li (Gordon) Tai created FLINK-4627:
--

 Summary: Use Flink's PropertiesUtil in Kinesis connector to 
extract typed values from config properties 
 Key: FLINK-4627
 URL: https://issues.apache.org/jira/browse/FLINK-4627
 Project: Flink
  Issue Type: Improvement
  Components: Kinesis Connector
Reporter: Tzu-Li (Gordon) Tai
Priority: Trivial
 Fix For: 1.2.0


Right now value extraction from config properties in the Kinesis connector is 
using the plain methods from {{java.util.Properties}} with type casting.

We can use Flink's utility {{PropertiesUtil}} to do this, with lesser lines of 
and more readable code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Gelly Library. Need an example

2016-09-15 Thread Vasiliki Kalavri 
Hi,

thanks for looking into this Till! I'm not quite sure what the algorithm
behavior should be when the vertex value is null (probably skip the
record?). Let's wait for Martin's input.

Cheers,
-V.

On 15 September 2016 at 19:19, Olga Golovneva  wrote:

> Hi Till,
>
> Thanks a lot for your help! I'll try to use another variable type in the
> meantime.
>
> Best regards,
> Olga
>
>
> Best regards,
> Olga Golovneva
>
> On Thu, Sep 15, 2016 at 1:03 PM, Till Rohrmann 
> wrote:
>
> > Hi Olga,
> >
> > it’s indeed an error in Flink’s Summarization algorithm. The problem is
> the
> > following: The vertex group value of the VertexGroupItem is null in the
> > VertexGroupReducer. This works in the SummarizationIT case because the
> > vertex value is of type String and the StringSerializer can deal with
> null
> > values.
> >
> > However, in your case where you use longs, it fails, because the
> > LongSerializer cannot handle null values. You can verify this behaviour
> by
> > changing the vertex value type to String. Then everything should work
> > without a problem.
> >
> > I’ve cc’ed Martin who can tell you probably more about the Summarization
> > algorithm. I’ve also opened a JIRA ticket [1] to fix this problem.
> >
> > Thanks for reporting this bug.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-4624
> >
> > Cheers,
> > Till
> > ​
> >
> > On Thu, Sep 15, 2016 at 5:05 PM, Olga Golovneva 
> > wrote:
> >
> > > Hi Till,
> > >
> > > I've created a simple (Java) example to show you what's going on. The
> > code
> > > is in attachment and shown below. This example creates simple graph
> with
> > > Double EV and Long VV. Then it runs Summarization, that should compute
> a
> > > condensed version of the input graph by grouping vertices and edges
> based
> > > on their values. I run this code with IntelliJ IDEA. The code executes
> > fine
> > > until you want to see what is written in resulted edges (just uncomment
> > > line 46, edgesOut.print();). Then it throws the following Exception:
> > >
> > > _EXCEPTION START_
> > > Exception in thread "main" org.apache.flink.runtime.
> > client.JobExecutionException:
> > > Job execution failed.
> > > at org.apache.flink.runtime.jobmanager.JobManager$$
> > > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$
> > > mcV$sp(JobManager.scala:830)
> > > at org.apache.flink.runtime.jobmanager.JobManager$$
> > > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(
> > JobManager.scala:773)
> > > at org.apache.flink.runtime.jobmanager.JobManager$$
> > > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(
> > JobManager.scala:773)
> > > at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> > > liftedTree1$1(Future.scala:24)
> > > at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> > > Future.scala:24)
> > > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> > > at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(
> > > AbstractDispatcher.scala:401)
> > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
> > > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> > > pollAndExecAll(ForkJoinPool.java:1253)
> > > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> > > runTask(ForkJoinPool.java:1346)
> > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> > > ForkJoinPool.java:1979)
> > > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> > > ForkJoinWorkerThread.java:107)
> > > Caused by: org.apache.flink.types.NullFieldException: Field 2 is null,
> > > but expected to hold a value.
> > > at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.serialize(
> > > TupleSerializer.java:126)
> > > at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.serialize(
> > > TupleSerializer.java:30)
> > > at org.apache.flink.runtime.plugable.SerializationDelegate.write(
> > > SerializationDelegate.java:56)
> > > at org.apache.flink.runtime.io.network.api.serialization.
> > > SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:83)
> > > at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(
> > > RecordWriter.java:85)
> > > at org.apache.flink.runtime.operators.shipping.
> OutputCollector.collect(
> > > OutputCollector.java:65)
> > > at org.apache.flink.runtime.operators.util.metrics.
> > > CountingCollector.collect(CountingCollector.java:35)
> > > at org.apache.flink.api.java.operators.translation.PlanFilterOperator$
> > > FlatMapFilter.flatMap(PlanFilterOperator.java:51)
> > > at org.apache.flink.runtime.operators.chaining.
> > > ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:80)
> > > at org.apache.flink.runtime.operators.util.metrics.
> > > CountingCollector.collect(CountingCollector.java:35)
> > > at org.apache.flink.graph.library.Summarization$
> > VertexGroupReducer.reduce(
> > > Summarization.java:323)
> > > at

Re: Gelly Library. Need an example

2016-09-15 Thread Till Rohrmann 
Hi Olga,

it’s indeed an error in Flink’s Summarization algorithm. The problem is the
following: The vertex group value of the VertexGroupItem is null in the
VertexGroupReducer. This works in the SummarizationIT case because the
vertex value is of type String and the StringSerializer can deal with null
values.

However, in your case where you use longs, it fails, because the
LongSerializer cannot handle null values. You can verify this behaviour by
changing the vertex value type to String. Then everything should work
without a problem.

I’ve cc’ed Martin who can tell you probably more about the Summarization
algorithm. I’ve also opened a JIRA ticket [1] to fix this problem.

Thanks for reporting this bug.

[1] https://issues.apache.org/jira/browse/FLINK-4624

Cheers,
Till
​

On Thu, Sep 15, 2016 at 5:05 PM, Olga Golovneva  wrote:

> Hi Till,
>
> I've created a simple (Java) example to show you what's going on. The code
> is in attachment and shown below. This example creates simple graph with
> Double EV and Long VV. Then it runs Summarization, that should compute a
> condensed version of the input graph by grouping vertices and edges based
> on their values. I run this code with IntelliJ IDEA. The code executes fine
> until you want to see what is written in resulted edges (just uncomment
> line 46, edgesOut.print();). Then it throws the following Exception:
>
> _EXCEPTION START_
> Exception in thread "main" 
> org.apache.flink.runtime.client.JobExecutionException:
> Job execution failed.
> at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$
> mcV$sp(JobManager.scala:830)
> at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:773)
> at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:773)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> liftedTree1$1(Future.scala:24)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(
> AbstractDispatcher.scala:401)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> pollAndExecAll(ForkJoinPool.java:1253)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1346)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
> Caused by: org.apache.flink.types.NullFieldException: Field 2 is null,
> but expected to hold a value.
> at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(
> TupleSerializer.java:126)
> at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(
> TupleSerializer.java:30)
> at org.apache.flink.runtime.plugable.SerializationDelegate.write(
> SerializationDelegate.java:56)
> at org.apache.flink.runtime.io.network.api.serialization.
> SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:83)
> at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(
> RecordWriter.java:85)
> at org.apache.flink.runtime.operators.shipping.OutputCollector.collect(
> OutputCollector.java:65)
> at org.apache.flink.runtime.operators.util.metrics.
> CountingCollector.collect(CountingCollector.java:35)
> at org.apache.flink.api.java.operators.translation.PlanFilterOperator$
> FlatMapFilter.flatMap(PlanFilterOperator.java:51)
> at org.apache.flink.runtime.operators.chaining.
> ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:80)
> at org.apache.flink.runtime.operators.util.metrics.
> CountingCollector.collect(CountingCollector.java:35)
> at org.apache.flink.graph.library.Summarization$VertexGroupReducer.reduce(
> Summarization.java:323)
> at org.apache.flink.runtime.operators.GroupReduceDriver.
> run(GroupReduceDriver.java:131)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:486)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:351)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:590)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.apache.flink.api.common.typeutils.base.LongSerializer.
> serialize(LongSerializer.java:64)
> at org.apache.flink.api.common.typeutils.base.LongSerializer.
> serialize(LongSerializer.java:27)
> at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(
> TupleSerializer.java:124)
> ... 15 more
>
> _EXCEPTION END__
>
> It looks like the problem is in the following lines in Summarization:
>
> DataSet> edgesForGrouping = input.getEdges()
>   .join(vertexToRepresentativeMap)
>   .where(0)  //

[jira] [Created] (FLINK-4624) Gelly's summarization algorithm cannot deal with null vertex group values

2016-09-15 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-4624:


 Summary: Gelly's summarization algorithm cannot deal with null 
vertex group values
 Key: FLINK-4624
 URL: https://issues.apache.org/jira/browse/FLINK-4624
 Project: Flink
  Issue Type: Bug
  Components: Gelly
Reporter: Till Rohrmann
 Fix For: 1.2.0


Gelly's {{Summarization}} algorithm cannot handle null values in the 
`VertexGroupItem.f2`. This behaviour is hidden by using Strings as a vertex 
value in the {{SummarizationITCase}}, because the {{StringSerializer}} can 
handle null values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flink HA on AWS: Network related issue

2016-09-15 Thread Deepak Jha 
Hi Till,
There is a way to shutdown actor systems by
setting  taskmanager.maxRegistrationDuration to a reasonable duration
(eg: 900 seconds). Default value sets it to Inf. In this case I noticed
that Taskmanager goes down and runit restarts the service and it gets
connected with Jobmanager.

 As I said earlier as well that retries to connect to Jobmanager does not
work even though telnet works at the same time to the same Jobmanager on
port 50050.  So retry does cache something which does not allow it to
reconnect. My flink cluster is on aws ( m4.large instances), not sure if
anyone else has observed this behavior.



On Thursday, September 15, 2016, Till Rohrmann  wrote:

> Hi Deepak,
>
> it seems that the JobManager's deathwatch declares the TaskManager to be
> unreachable which will automatically quarantine it. You're right that in
> such a case, the TaskManager should shut down and be restarted so that it
> can again reconnect to the JobManager. This is, however, not yet supported
> automatically.
>
> For the moment, I'd recommend you to make the deathwatch a little bit less
> aggressive via the following settings:
>
> - akka.watch.heartbeat.interval: Heartbeat interval for Akka’s DeathWatch
> mechanism to detect dead TaskManagers. If TaskManagers are wrongly marked
> dead because of lost or delayed heartbeat messages, then you should
> increase this value. A thorough description of Akka’s DeathWatch can be
> found here (DEFAULT: akka.ask.timeout/10).
> - akka.watch.heartbeat.pause: Acceptable heartbeat pause for Akka’s
> DeathWatch mechanism. A low value does not allow a irregular heartbeat. A
> thorough description of Akka’s DeathWatch can be found here (DEFAULT:
> akka.ask.timeout).
> - akka.watch.threshold: Threshold for the DeathWatch failure detector. A
> low value is prone to false positives whereas a high value increases the
> time to detect a dead TaskManager. A thorough description of Akka’s
> DeathWatch can be found here (DEFAULT: 12).
>
> I hope this helps you to work around the problem for the moment until we've
> added the automatic shut down and restart.
>
> Cheers,
> Till
>
> On Mon, Sep 12, 2016 at 5:55 AM, Deepak Jha  > wrote:
>
> > Hi Till,
> > One more thing i noticed after looking into following message in
> > taskmanager log
> >
> > 2016-09-11 17:57:25,310 PDT [WARN]  ip-10-6-0-15
> > [flink-akka.actor.default-dispatcher-31] Remoting - Tried to associate
> > with
> > unreachable remote address [akka.tcp://flink@10.6.22.22:50050]. Address
> is
> > now gated for 5000 ms, all messages to this address will be delivered to
> > dead letters. Reason: *The remote system has quarantined this system. No
> > further associations to the remote system are possible until this system
> is
> > restarted*.
> >
> > So in this case ideally the local ActorSystem should go down so that
> > service supervisor/runit will restart the system and taskmanager will
> again
> > be able to connect to the remote system.. If it does not happen
> > automatically then we have to monitor logs in some way and then try to
> > ensure that it restarts. Ideally flink taskmanager Actor System should go
> > down. Please let me know if my understanding is wrong.
> >
> >
> >
> >
> >
> > On Fri, Sep 9, 2016 at 8:01 AM, Deepak Jha  > wrote:
> >
> > > Hi Till,
> > > I'm getting following message in Jobmanager log
> > >
> > > 2016-09-09 07:46:55,093 PDT [WARN]  ip-10-8-11-249
> > > [flink-akka.actor.default-dispatcher-985] akka.remote.RemoteWatcher -
> > *Detected
> > > unreachable: [akka.tcp://flink@10.8.4.57:6121
> > > ]*
> > > 2016-09-09 07:46:55,094 PDT [INFO]  ip-10-8-11-249
> > > [flink-akka.actor.default-dispatcher-985] o.a.f.runtime.jobmanager.
> > JobManager
> > > - Task manager akka.tcp://flink@10.8.4.57:6121/user/taskmanager
> > > terminated.
> > > 2016-09-09 07:46:55,094 PDT [INFO]  ip-10-8-11-249
> > > [flink-akka.actor.default-dispatcher-985] o.a.f.r.instance.
> > InstanceManager
> > > - Unregistered task manager akka.tcp://flink@10.8.4.57 :
> > > 6121/user/taskmanager. Number of registered task managers 2. Number of
> > > available slots 4.
> > > 2016-09-09 07:46:55,096 PDT [WARN]  ip-10-8-11-249
> > > [flink-akka.actor.default-dispatcher-982] Remoting - Association to
> > > [akka.tcp://flink@10.8.4.57:6121] having UID [-1223410403] is
> > > irrecoverably failed. *UID is now quarantined and all messages to this
> > > UID will be delivered to dead letters. Remote actorsystem must be
> > restarted
> > > to recover from this situation.*
> > > 2016-09-09 07:46:55,097 PDT [INFO]  ip-10-8-11-249
> > > [flink-akka.actor.default-dispatcher-982] akka.actor.LocalActorRef -
> > > Message [akka.remote.transport.AssociationHandle$Disassociated] from
> > > Actor[akka://flink/deadLetters] to Actor[akka://flink/system/
> > > endpointManager/reliableEndpointWriter-akka.
> >

[jira] [Created] (FLINK-4623) Create Physical Execution Plan of a DataStream

2016-09-15 Thread Timo Walther (JIRA) 
Timo Walther created FLINK-4623:
---

 Summary: Create Physical Execution Plan of a DataStream
 Key: FLINK-4623
 URL: https://issues.apache.org/jira/browse/FLINK-4623
 Project: Flink
  Issue Type: Improvement
  Components: Table API & SQL
Reporter: Timo Walther


The {{StreamTableEnvironment#explain(Table)}} command for tables of a 
{{StreamTableEnvironment}} only outputs the abstract syntax tree. It would be 
helpful if the {{explain}} method could also generate a string from the 
{{DataStream}} containing a physical execution plan.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: build error

2016-09-15 Thread Ivan Mushketyk 
Hi Ameet,

The new issue that you are facing is caused by a build-time validator that
checks if every file in Flink sources has an Apache license text in it.
It suggests that there are two files without a proper license:
* build_log_maven_flink
* log.oracle.jdk

If you remove them or move from the Flink source folder this error will go
away.

Best regards,
Ivan.

On Thu, 15 Sep 2016 at 15:57 Ameet BD  wrote:

> Hi Ivan,
>
> Thanks for reply. I tried installing Oracle JDK. The build error log is
> attached for details.
> Error reported is "Too many files with unapproved license".
> The file rat.txt is attached.
>
> Please tell me what could be going wrong. I am using Ubuntu 14.04, Maven
> 3.0.5 and Oracle JDK 8.
> Regards,
> Ameet
>
> On Thu, Sep 15, 2016 at 5:45 PM, Ivan Mushketyk 
> wrote:
>
>> Hi Ameet,
>>
>> This seems like a possible JVM bug.
>> Could you try to build Flink using OracleJDK instead?
>>
>> Best regards,
>> Ivan.
>>
>> On Thu, 15 Sep 2016 at 12:30 Ameet BD  wrote:
>>
>> > Hi All,
>> >
>> > I am trying to build latest Flink code from github.com by running
>> command
>> >
>> > $ git clone https://github.com/apache/flink
>> > After that I am running command
>> > $ mvn clean install -DskipTests
>> >
>> > The build fails I have attached the build log.
>> >
>> > Kindly let me know why it may be failing.
>> >
>> > Regards.
>> > Ameet
>> >
>>
>
>

Re: Gelly Library. Need an example

2016-09-15 Thread Olga Golovneva 
Hi Till,

I've created a simple (Java) example to show you what's going on. The code
is in attachment and shown below. This example creates simple graph with
Double EV and Long VV. Then it runs Summarization, that should compute a
condensed version of the input graph by grouping vertices and edges based
on their values. I run this code with IntelliJ IDEA. The code executes fine
until you want to see what is written in resulted edges (just uncomment
line 46, edgesOut.print();). Then it throws the following Exception:

_EXCEPTION START_
Exception in thread "main"
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$mcV$sp(JobManager.scala:830)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:773)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:773)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.types.NullFieldException: Field 2 is null, but
expected to hold a value.
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:126)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:30)
at
org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:83)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:85)
at
org.apache.flink.runtime.operators.shipping.OutputCollector.collect(OutputCollector.java:65)
at
org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
at
org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:51)
at
org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:80)
at
org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
at
org.apache.flink.graph.library.Summarization$VertexGroupReducer.reduce(Summarization.java:323)
at
org.apache.flink.runtime.operators.GroupReduceDriver.run(GroupReduceDriver.java:131)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:486)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:351)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:590)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.flink.api.common.typeutils.base.LongSerializer.serialize(LongSerializer.java:64)
at
org.apache.flink.api.common.typeutils.base.LongSerializer.serialize(LongSerializer.java:27)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:124)
... 15 more

_EXCEPTION END__

It looks like the problem is in the following lines in Summarization:

DataSet> edgesForGrouping = input.getEdges()
  .join(vertexToRepresentativeMap)
  .where(0)  // source vertex id
  .equalTo(0) // vertex id
  .with(new SourceVertexJoinFunction())
  .join(vertexToRepresentativeMap)
  .where(1)  // target vertex id
  .equalTo(0) // vertex id
  .with(new TargetVertexJoinFunction());


If you try to print edges before this step, it works fine. But after this
step my IDE gives the same exception.

I would really appreciate any help.

Thank you,
Olga

_EXAMPLE START_

package org.apache.flink.graph.examples;

import org.apache.flink.api.common.ProgramDescription;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.graph.Edge;
import org.apache.flink.graph.Graph;
import org.apache.flink.graph.Vertex;
import org.apache.flink.graph.library.Summarization;
import java.util.LinkedList;
import java.util.List;

public class MySummarizationExample implements ProgramDescription {

@SuppressWarnings("serial")
public static void main(String [] args) throws Exception {

[jira] [Created] (FLINK-4622) CLI help message should include 'savepoint' action

2016-09-15 Thread Scott Kidder (JIRA) 
Scott Kidder created FLINK-4622:
---

 Summary: CLI help message should include 'savepoint' action
 Key: FLINK-4622
 URL: https://issues.apache.org/jira/browse/FLINK-4622
 Project: Flink
  Issue Type: Bug
  Components: Client
Affects Versions: 1.1.2
Reporter: Scott Kidder
Assignee: Scott Kidder
Priority: Trivial


The Flink CLI help message should include the 'savepoint' action in the list of 
available actions. It currently looks like:

{code}
bash-4.3# flink foo
"foo" is not a valid action.

Valid actions are "run", "list", "info", "stop", or "cancel".

Specify the version option (-v or --version) to print Flink version.

Specify the help option (-h or --help) to get help on the command.
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: build error

2016-09-15 Thread Till Rohrmann 
I agree with Ivan. This does not look like a Flink specific problem. Have
you tried building other projects with your maven and JDK version?

Cheers,
Till

On Thu, Sep 15, 2016 at 2:15 PM, Ivan Mushketyk 
wrote:

> Hi Ameet,
>
> This seems like a possible JVM bug.
> Could you try to build Flink using OracleJDK instead?
>
> Best regards,
> Ivan.
>
> On Thu, 15 Sep 2016 at 12:30 Ameet BD  wrote:
>
> > Hi All,
> >
> > I am trying to build latest Flink code from github.com by running
> command
> >
> > $ git clone https://github.com/apache/flink
> > After that I am running command
> > $ mvn clean install -DskipTests
> >
> > The build fails I have attached the build log.
> >
> > Kindly let me know why it may be failing.
> >
> > Regards.
> > Ameet
> >
>

Re: Gelly Library. Need an example

2016-09-15 Thread Till Rohrmann 
Hi Olga,

can you provide us with a little bit more details about the problem. The
full stack trace of the exception and the program you're trying to run
would be helpful.

Cheers,
Till

On Wed, Sep 14, 2016 at 9:49 PM, Olga Golovneva  wrote:

> Hi devs,
>
> Do you know if there is an example (besides ITCase) of usage of
> Summarization Library in Gelly? I'm having some problems trying to use it
> in my code. Particularly, I cannot print output edges ( it throws the
> following exception: Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException: Job execution
> failed.), while vertices are printed correctly.
>
> Best regards,
> Olga
>

Re: build error

2016-09-15 Thread Ivan Mushketyk 
Hi Ameet,

This seems like a possible JVM bug.
Could you try to build Flink using OracleJDK instead?

Best regards,
Ivan.

On Thu, 15 Sep 2016 at 12:30 Ameet BD  wrote:

> Hi All,
>
> I am trying to build latest Flink code from github.com by running command
>
> $ git clone https://github.com/apache/flink
> After that I am running command
> $ mvn clean install -DskipTests
>
> The build fails I have attached the build log.
>
> Kindly let me know why it may be failing.
>
> Regards.
> Ameet
>

Re: ML contributions

2016-09-15 Thread Theodore Vasiloudis 
That's great to hear Gabor, I'll definitely help out with the review
process, and I hope we can get some committer to look into these and other
outstanding PRs for FlinkML.

On Thu, Sep 15, 2016 at 11:59 AM, Till Rohrmann 
wrote:

> Great to hear Gabor :-) I hope that the community will help out with
> reviewing of the algorithms so that we can give quick feedback. Looking
> forward to your contributions.
>
> Cheers,
> Till
>
> On Mon, Sep 12, 2016 at 10:34 AM, Gábor Hermann 
> wrote:
>
> > Hey all,
> >
> > We are planning to contribute some algorithms and improvements to Flink
> ML
> > at SZTAKI .
> > I have already opened a JIRA  > a/browse/FLINK-4613> for an implicit feedback ALS, but probably more will
> > come soon.
> >
> > We are implementing algorithms anyway, and it would be nice if we could
> > incorporate them into Flink,
> > so I just wanted to let you all know about our efforts.
> >
> > Cheers,
> > Gabor
> >
>

build error

2016-09-15 Thread Ameet BD 
Hi All,

I am trying to build latest Flink code from github.com by running command

$ git clone https://github.com/apache/flink
After that I am running command
$ mvn clean install -DskipTests

The build fails I have attached the build log.

Kindly let me know why it may be failing.

Regards.
Ameet
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fa3b34ea3c7, pid=10719, tid=140341028869888
#
# JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x48b3c7]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---  T H R E A D  ---

Current thread (0x7fa3ac01f800):  GCTaskThread [stack: 0x7fa3b11c5000,0x7fa3b12c6000] [id=10723]

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0xe573c300

Registers:
RAX=0x7fa3b3effac0, RBX=0xe573c2f8, RCX=0x001f, RDX=0x8000
RSP=0x7fa3b12c4bc0, RBP=0x7fa3b12c4bc0, RSI=0x0001, RDI=0xe573c2f8
R8 =0xe573c2f8, R9 =0x0011, R10=0x7fa3b3f29080, R11=0x0080
R12=0x7fa3ac074890, R13=0x0001001d2a08, R14=0xe089c124, R15=0xe089c138
RIP=0x7fa3b34ea3c7, EFLAGS=0x00010202, CSGSFS=0x0033, ERR=0x0004
  TRAPNO=0x000e

Top of Stack: (sp=0x7fa3b12c4bc0)
0x7fa3b12c4bc0:   7fa3b12c4c30 7fa3b36598f0
0x7fa3b12c4bd0:   0010 7fa3ac0748d0
0x7fa3b12c4be0:   e089c100 0001001d2a10
0x7fa3b12c4bf0:   7fa3b3f29080 0003
0x7fa3b12c4c00:   7fa3ac027340 7fa3b3f250f0
0x7fa3b12c4c10:   7fa3ac0747a0 7fa3ac074890
0x7fa3b12c4c20:   7fa3b12c4d6c 7fa3b12c4d70
0x7fa3b12c4c30:   7fa3b12c4cf0 7fa3b3915de2
0x7fa3b12c4c40:   7fa3ac0748a0 7fa3ac074920
0x7fa3b12c4c50:   7fa3ac074950 e1b33e20
0x7fa3b12c4c60:   7fa3ac0748a0 e20b0e50
0x7fa3b12c4c70:   7fa3ac0748d0 7fa3ac074890
0x7fa3b12c4c80:   7fa3b12c4cf0 7fa3b3659927
0x7fa3b12c4c90:   e089c100 7fa3ac0748d0
0x7fa3b12c4ca0:   001e001b 0001002da968
0x7fa3b12c4cb0:   001f001b 0005
0x7fa3b12c4cc0:   7fa3ac01f800 7fa3ac074890
0x7fa3b12c4cd0:   7fa3ac0747a0 
0x7fa3b12c4ce0:   7fa3b12c4d6c 7fa3b12c4d70
0x7fa3b12c4cf0:   7fa3b12c4dc0 7fa3b38faa2e
0x7fa3b12c4d00:   7fa3ac074890 7fa3ac020298
0x7fa3b12c4d10:   7fa3b3ee52c0 7fa3ac0748d0
0x7fa3b12c4d20:   7fa3b12c4cf0 
0x7fa3b12c4d30:   7fa3b3eddb60 7fa3ac01f800
0x7fa3b12c4d40:   7fa3b12c4d80 7fa3b12c4d70
0x7fa3b12c4d50:   7fa3ac027640 7fa3ac074890
0x7fa3b12c4d60:    00b010d3ac01f170
0x7fa3b12c4d70:   e31a3d30 7fa3b387d61b
0x7fa3b12c4d80:    7fa3
0x7fa3b12c4d90:   7fa3b12c4dc0 0001
0x7fa3b12c4da0:   7fa3ac027640 7fa3b3ac6775
0x7fa3b12c4db0:   7fa3ac01f800 7fa3b3f1e3b8 

Instructions: (pc=0x7fa3b34ea3c7)
0x7fa3b34ea3a7:   8b 42 08 83 f8 00 7e 2c a8 01 75 2d c1 f8 03 5d
0x7fa3b34ea3b7:   c3 0f 1f 84 00 00 00 00 00 48 8d 05 f9 56 a1 00
0x7fa3b34ea3c7:   8b 57 08 8b 48 08 48 d3 e2 48 03 10 8b 42 08 83
0x7fa3b34ea3d7:   f8 00 7f d4 75 1b 0f 1f 00 48 8b 02 48 89 fe 48 

Register to memory mapping:

RAX=0x7fa3b3effac0:  in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so at 0x7fa3b305f000
RBX=0xe573c2f8 is an unallocated location in the heap
RCX=0x001f is an unknown value
RDX=0x8000 is an unknown value
RSP=0x7fa3b12c4bc0 is an unknown value
RBP=0x7fa3b12c4bc0 is an unknown value
RSI=0x0001 is an unknown value
RDI=0xe573c2f8 is an unallocated location in the heap
R8 =0xe573c2f8 is an unallocated location in the heap
R9 =0x0011 is an unknown value
R10=0x7fa3b3f29080:  in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so at 0x7fa3b305f000
R11=0x0080 is an unknown value
R12=0x7fa3ac074890 is an unknown value
R13=0x0001001d2a08 is pointing into metadata
R14=0xe089c124 is pointing into object: 0xe089c100
scala.reflect.io.FileZipArchive$FileEntry$1 
 - klass: 'scala/reflect/io/FileZipArchive$FileEntry$1'
R15=0xe089c138 is an oop
java.lang.String 
 - klass: 'java/lang/String'


Stack: [0x7fa3b11c5000,0x7fa3b12c6000],

Re: Savepoint and restart in the same command?

2016-09-15 Thread Till Rohrmann 
Hi Ron,

Konstantin is absolutely right. One of the missing features is an atomic
stop with savepoint operation to shut down a job in a clean manner. We will
hopefully have this soon.

Cheers,
Till

On Tue, Sep 13, 2016 at 11:53 AM, Konstantin Knauf <
konstantin.kn...@tngtech.com> wrote:

> Hi Ron,
>
> I think, Till mentioned in his talk on re-scaling streaming jobs, that a
> "Savepoint and Stop" command is one of the upcoming features in the
> "dynamic scaling" thread.
>
> Cheers,
>
> Konstantin
>
> On 12.09.2016 14:39, Ron Crocker wrote:
> > I attended Jamie Grier’s excellent presentation at Flink Forward today
> ("Robust Stream Processing with Apache Flink” -
> http://flink-forward.org/kb_sessions/robust-stream-
> processing-with-apache-flink/  sessions/robust-stream-processing-with-apache-flink/>) where he showed
> recovery from a savepoint. It took at least 3 steps - create the savepoint,
> kill the current job, start from savepoint. Is it possible to combine these
> into a single action - not just from the tooling perspective (where I think
> it’s pretty straightforward to simulate what I’m asking for here) but to
> actually have the running job STOP once it processes the save point request
> AND start a new job from that savepoint?
> >
> > What I’m looking to reduce the gap between the savepoint creation AND
> the stopping of the current task as well as reducing the gap between
> stopping the current job and starting a new job from the savepoint.
> >
> > Ron
> > —
> > Ron Crocker
> > Principal Engineer & Architect
> > ( ( •)) New Relic
> > rcroc...@newrelic.com
> > M: +1 630 363 8835
> >
> >
>
> --
> Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

Re: ML contributions

2016-09-15 Thread Till Rohrmann 
Great to hear Gabor :-) I hope that the community will help out with
reviewing of the algorithms so that we can give quick feedback. Looking
forward to your contributions.

Cheers,
Till

On Mon, Sep 12, 2016 at 10:34 AM, Gábor Hermann 
wrote:

> Hey all,
>
> We are planning to contribute some algorithms and improvements to Flink ML
> at SZTAKI .
> I have already opened a JIRA  a/browse/FLINK-4613> for an implicit feedback ALS, but probably more will
> come soon.
>
> We are implementing algorithms anyway, and it would be nice if we could
> incorporate them into Flink,
> so I just wanted to let you all know about our efforts.
>
> Cheers,
> Gabor
>

[jira] [Created] (FLINK-4621) Improve decimal literals of SQL API

2016-09-15 Thread Timo Walther (JIRA) 
Timo Walther created FLINK-4621:
---

 Summary: Improve decimal literals of SQL API
 Key: FLINK-4621
 URL: https://issues.apache.org/jira/browse/FLINK-4621
 Project: Flink
  Issue Type: Improvement
  Components: Table API & SQL
Reporter: Timo Walther


Currently, all SQL {{DECIMAL}} types are converted to BigDecimals internally. 
By default, the SQL parsers creates {{DECIMAL}} literals of any number e.g. 
{{SELECT 1.0, 12, -0.5 FROM x}}. I think it would be better if these simple 
numbers would be represented as Java primitives instead of objects.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)