[jira] [Created] (FLINK-3968) Cancel button on a running job not available on smaller screens

[Lokesh Ravindranathan (JIRA)]

Lokesh Ravindranathan created FLINK-3968:


 Summary: Cancel button on a running job not available on smaller 
screens
 Key: FLINK-3968
 URL: https://issues.apache.org/jira/browse/FLINK-3968
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Affects Versions: 1.0.1
Reporter: Lokesh Ravindranathan
Priority: Minor


I am using a Mac with Chrome as my browser. I run a Flink job and when I look 
at the dashboard, there is no cancel button and I cannot scroll the page to the 
right on my Mac display(13"). But when I connect to a second monitor (24th), i 
can see the button visible. I presume this should be a rendering problem. The 
two screenshots one having the cancel button and the other missing the same - 
http://imgur.com/a/1nTSg.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3967) Provide RethinkDB Sink for Flink

[Mans Singh (JIRA)]

Mans Singh created FLINK-3967:
-

 Summary: Provide RethinkDB Sink for Flink
 Key: FLINK-3967
 URL: https://issues.apache.org/jira/browse/FLINK-3967
 Project: Flink
  Issue Type: New Feature
  Components: Streaming, Streaming Connectors
Affects Versions: 1.0.3
 Environment: All
Reporter: Mans Singh
Assignee: Mans Singh
Priority: Minor
 Fix For: 1.1.0


Provide Sink to stream data from flink to rethink db.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3966) AvroTypeInfo does not work with readonly avro specific records

[Matthew Burghoffer (JIRA)]

Matthew Burghoffer created FLINK-3966:
-

 Summary: AvroTypeInfo does not work with readonly avro specific 
records
 Key: FLINK-3966
 URL: https://issues.apache.org/jira/browse/FLINK-3966
 Project: Flink
  Issue Type: Bug
  Components: Avro Support
Affects Versions: 1.0.2, 1.0.3
Reporter: Matthew Burghoffer
Priority: Minor


When generating avro code, users will often supply createSetters=false and 
fieldVisibility=private for Avro specific records (for effectively immutable 
records).  This means that, according to TypeExtractor.isValidPojoField, the 
object is not a valid pojo (though is completely a valid avro object) and the 
TypeInformation factory will fail.

Specific records are usually created using builders, so perhaps using this 
mechanism for when creating a new record / mutating an existing record is 
required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3965) Delegating GraphAlgorithm

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-3965:
-

 Summary: Delegating GraphAlgorithm
 Key: FLINK-3965
 URL: https://issues.apache.org/jira/browse/FLINK-3965
 Project: Flink
  Issue Type: New Feature
  Components: Gelly
Affects Versions: 1.1.0
Reporter: Greg Hogan
Assignee: Greg Hogan
 Fix For: 1.1.0


Complex and related algorithms often overlap in computation of data. Two such 
examples are:
1) the local and global clustering coefficients each use a listing of triangles
2) the local clustering coefficient joins on vertex degree, and the underlying 
triangle listing annotates edge degree which uses vertex degree

We can reuse and rewrite algorithm output by creating a {{ProxyObject}} as a 
delegate for method calls to the {{DataSet}} returned by the algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Blogpost on Flink's SQL support

[Fabian Hueske]

Thanks Henry! The post is out :-)

2016-05-24 16:51 GMT+02:00 Henry Saputra :

> Awesome!
> Thanks for the great post, Fabian
>
> - Henry
>
> On Tue, May 24, 2016 at 2:36 AM, Fabian Hueske  wrote:
>
> > Thanks everybody for the feedback and comments!
> >
> > I moved the Google doc into Markdown and opened a PR:
> > https://github.com/apache/flink-web/pull/22
> >
> > Will merge this PR and publish the post later today.
> >
> > Thanks, Fabian
> >
> > 2016-05-24 10:01 GMT+02:00 Kostas Tzoumas :
> >
> > > +1, great post
> > >
> > > On Sun, May 22, 2016 at 4:15 PM, Matthias J. Sax 
> > wrote:
> > >
> > > > Will be a nice post!
> > > >
> > > > On 05/21/2016 10:40 PM, Henry Saputra wrote:
> > > > > I agree with Ufuk, that this is more internal posts which perfect
> for
> > > > blog.
> > > > >
> > > > > For high level and use cases I think would be better to be added to
> > > > Apache
> > > > > Flink release docs bc that is where most users will try to find
> info
> > on
> > > > how
> > > > > to use it.
> > > > >
> > > > > - Henry
> > > > >
> > > > > On Saturday, May 21, 2016, Ufuk Celebi  wrote:
> > > > >
> > > > >> Hey Fabian,
> > > > >>
> > > > >> thank you for this blog post. I added some minor comments in the
> > > > >> document. Great read and great work by you and the others who have
> > > > >> contributed to SQL! :-)
> > > > >>
> > > > >> In general, I think that the post is very much an "Internals" post
> > > > >> like the "bits and bytes" one. This is definitely nice, but I
> think
> > > > >> that we should definitely follow up with a high-level/use case
> > driven
> > > > >> post after the release.
> > > > >>
> > > > >> – Ufuk
> > > > >>
> > > > >>
> > > > >> On Sat, May 21, 2016 at 5:07 PM, Aljoscha Krettek <
> > > aljos...@apache.org
> > > > >> > wrote:
> > > > >>> A great post!
> > > > >>>
> > > > >>> I had some small comments on the doc.
> > > > >>>
> > > > >>> On Sat, 21 May 2016 at 16:52 Robert Metzger  > > > >> > wrote:
> > > > >>>
> > > >  Thanks a lot for the great blog post!
> > > > 
> > > >  +1 for publishing it on the Flink blog.
> > > > 
> > > >  On Fri, May 20, 2016 at 5:12 PM, Fabian Hueske <
> fhue...@gmail.com
> > > > >> > wrote:
> > > > 
> > > > > Hi everybody,
> > > > >
> > > > > I wrote a blog post about the SQL efforts of the Flink
> community
> > > and
> > > >  would
> > > > > like to get your feedback.
> > > > >
> > > > > You can read and comment the Google doc:
> > > > >
> > > > >
> > > > >
> > > > 
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1xy6d9w6Gjm8Bsh9SJbGuTZsulBJxmdIXhESJ4dV2jEY
> > > > >
> > > > > I am planning to publish the post around next Tuesday.
> > > > >
> > > > > Looking forward to your comments and have a nice weekend,
> > > > > Fabian
> > > > >
> > > > 
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-3964) Job submission times out with recursive.file.enumeration

[Juho Autio (JIRA)]

Juho Autio created FLINK-3964:
-

 Summary: Job submission times out with recursive.file.enumeration
 Key: FLINK-3964
 URL: https://issues.apache.org/jira/browse/FLINK-3964
 Project: Flink
  Issue Type: Bug
Reporter: Juho Autio


When using "recursive.file.enumeration" with a big enough folder structure to 
list, flink batch job fails right at the beginning because of a timeout.

h2. Problem details

We get this error: {{Communication with JobManager failed: Job submission to 
the JobManager timed out}}.

The code we have is basically this:

{code}
val env = ExecutionEnvironment.getExecutionEnvironment

val parameters = new Configuration

// set the recursive enumeration parameter
parameters.setBoolean("recursive.file.enumeration", true)

val parameter = ParameterTool.fromArgs(args)

val input_data_path : String = parameter.get("input_data_path", null )

val data : DataSet[(Text,Text)] = env.readSequenceFile(classOf[Text], 
classOf[Text], input_data_path)
.withParameters(parameters)

data.first(10).print
{code}

If we set {{input_data_path}} parameter to {{s3n://bucket/path/date=*/}} it 
times out. If we use a more restrictive pattern like 
{{s3n://bucket/path/date=20160523/}}, it doesn't time out.

To me it seems that time taken to list files shouldn't cause any timeouts on 
job submission level.

For us this was "fixed" by adding {{akka.client.timeout: 600 s}} in 
{{flink-conf.yaml}}, but I wonder if the timeout would still occur if we have 
even more files to list?



P.S. Is there any way to set {{akka.client.timeout}} when calling {{bin/flink 
run}} instead of editing {{flink-conf.yaml}}. I tried to add it as a {{-yD}} 
flag but couldn't get it working.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3963) AbstractReporter uses shaded dependency

[Maximilian Michels (JIRA)]

Maximilian Michels created FLINK-3963:
-

 Summary: AbstractReporter uses shaded dependency
 Key: FLINK-3963
 URL: https://issues.apache.org/jira/browse/FLINK-3963
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.1.0
Reporter: Maximilian Michels
Assignee: Kostas Kloudas
 Fix For: 1.1.0


This fails our Hadoop 1 build on Travis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3962) JMXReporter doesn't properly register/deregister metrics

[Maximilian Michels (JIRA)]

Maximilian Michels created FLINK-3962:
-

 Summary: JMXReporter doesn't properly register/deregister metrics
 Key: FLINK-3962
 URL: https://issues.apache.org/jira/browse/FLINK-3962
 Project: Flink
  Issue Type: Bug
  Components: TaskManager
Affects Versions: 1.1.0
Reporter: Maximilian Michels
Assignee: Chesnay Schepler
 Fix For: 1.1.0


The following fails our Yarn tests because it checks for errors in the 
jobmanager/taskmanager logs:

{noformat}
2016-05-23 19:20:02,349 ERROR org.apache.flink.metrics.reporter.JMXReporter 
- A metric with the name 
org.apache.flink.metrics:key0=testing-worker-linux-docker-05a6b382-3386-linux-4,key1=taskmanager,key2=9398ca9392af615e9d1896d0bd7ff52a,key3=Flink_Java_Job_at_Mon_May_23_19-20-00_UTC_2016,key4=,name=numBytesIn
 was already registered.
javax.management.InstanceAlreadyExistsException: 
org.apache.flink.metrics:key0=testing-worker-linux-docker-05a6b382-3386-linux-4,key1=taskmanager,key2=9398ca9392af615e9d1896d0bd7ff52a,key3=Flink_Java_Job_at_Mon_May_23_19-20-00_UTC_2016,key4=,name=numBytesIn
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at 
org.apache.flink.metrics.reporter.JMXReporter.notifyOfAddedMetric(JMXReporter.java:76)
at 
org.apache.flink.metrics.MetricRegistry.register(MetricRegistry.java:177)
at 
org.apache.flink.metrics.groups.AbstractMetricGroup.addMetric(AbstractMetricGroup.java:191)
at 
org.apache.flink.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:144)
at 
org.apache.flink.metrics.groups.IOMetricGroup.(IOMetricGroup.java:40)
at 
org.apache.flink.metrics.groups.TaskMetricGroup.(TaskMetricGroup.java:68)
at 
org.apache.flink.metrics.groups.JobMetricGroup.addTask(JobMetricGroup.java:74)
at 
org.apache.flink.metrics.groups.TaskManagerMetricGroup.addTaskForJob(TaskManagerMetricGroup.java:86)
at 
org.apache.flink.runtime.taskmanager.TaskManager.submitTask(TaskManager.scala:1092)
at 
org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleTaskMessage(TaskManager.scala:441)
at 
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:283)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at 
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at 
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at 
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:124)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Blogpost on Flink's SQL support

[Fabian Hueske]

Thanks everybody for the feedback and comments!

I moved the Google doc into Markdown and opened a PR:
https://github.com/apache/flink-web/pull/22

Will merge this PR and publish the post later today.

Thanks, Fabian

2016-05-24 10:01 GMT+02:00 Kostas Tzoumas :

> +1, great post
>
> On Sun, May 22, 2016 at 4:15 PM, Matthias J. Sax  wrote:
>
> > Will be a nice post!
> >
> > On 05/21/2016 10:40 PM, Henry Saputra wrote:
> > > I agree with Ufuk, that this is more internal posts which perfect for
> > blog.
> > >
> > > For high level and use cases I think would be better to be added to
> > Apache
> > > Flink release docs bc that is where most users will try to find info on
> > how
> > > to use it.
> > >
> > > - Henry
> > >
> > > On Saturday, May 21, 2016, Ufuk Celebi  wrote:
> > >
> > >> Hey Fabian,
> > >>
> > >> thank you for this blog post. I added some minor comments in the
> > >> document. Great read and great work by you and the others who have
> > >> contributed to SQL! :-)
> > >>
> > >> In general, I think that the post is very much an "Internals" post
> > >> like the "bits and bytes" one. This is definitely nice, but I think
> > >> that we should definitely follow up with a high-level/use case driven
> > >> post after the release.
> > >>
> > >> – Ufuk
> > >>
> > >>
> > >> On Sat, May 21, 2016 at 5:07 PM, Aljoscha Krettek <
> aljos...@apache.org
> > >> > wrote:
> > >>> A great post!
> > >>>
> > >>> I had some small comments on the doc.
> > >>>
> > >>> On Sat, 21 May 2016 at 16:52 Robert Metzger  > >> > wrote:
> > >>>
> >  Thanks a lot for the great blog post!
> > 
> >  +1 for publishing it on the Flink blog.
> > 
> >  On Fri, May 20, 2016 at 5:12 PM, Fabian Hueske  > >> > wrote:
> > 
> > > Hi everybody,
> > >
> > > I wrote a blog post about the SQL efforts of the Flink community
> and
> >  would
> > > like to get your feedback.
> > >
> > > You can read and comment the Google doc:
> > >
> > >
> > >
> > 
> > >>
> >
> https://docs.google.com/document/d/1xy6d9w6Gjm8Bsh9SJbGuTZsulBJxmdIXhESJ4dV2jEY
> > >
> > > I am planning to publish the post around next Tuesday.
> > >
> > > Looking forward to your comments and have a nice weekend,
> > > Fabian
> > >
> > 
> > >>
> > >
> >
> >
>

Re: Blogpost on Flink's SQL support

[Kostas Tzoumas]

+1, great post

On Sun, May 22, 2016 at 4:15 PM, Matthias J. Sax  wrote:

> Will be a nice post!
>
> On 05/21/2016 10:40 PM, Henry Saputra wrote:
> > I agree with Ufuk, that this is more internal posts which perfect for
> blog.
> >
> > For high level and use cases I think would be better to be added to
> Apache
> > Flink release docs bc that is where most users will try to find info on
> how
> > to use it.
> >
> > - Henry
> >
> > On Saturday, May 21, 2016, Ufuk Celebi  wrote:
> >
> >> Hey Fabian,
> >>
> >> thank you for this blog post. I added some minor comments in the
> >> document. Great read and great work by you and the others who have
> >> contributed to SQL! :-)
> >>
> >> In general, I think that the post is very much an "Internals" post
> >> like the "bits and bytes" one. This is definitely nice, but I think
> >> that we should definitely follow up with a high-level/use case driven
> >> post after the release.
> >>
> >> – Ufuk
> >>
> >>
> >> On Sat, May 21, 2016 at 5:07 PM, Aljoscha Krettek  >> > wrote:
> >>> A great post!
> >>>
> >>> I had some small comments on the doc.
> >>>
> >>> On Sat, 21 May 2016 at 16:52 Robert Metzger  >> > wrote:
> >>>
>  Thanks a lot for the great blog post!
> 
>  +1 for publishing it on the Flink blog.
> 
>  On Fri, May 20, 2016 at 5:12 PM, Fabian Hueske  >> > wrote:
> 
> > Hi everybody,
> >
> > I wrote a blog post about the SQL efforts of the Flink community and
>  would
> > like to get your feedback.
> >
> > You can read and comment the Google doc:
> >
> >
> >
> 
> >>
> https://docs.google.com/document/d/1xy6d9w6Gjm8Bsh9SJbGuTZsulBJxmdIXhESJ4dV2jEY
> >
> > I am planning to publish the post around next Tuesday.
> >
> > Looking forward to your comments and have a nice weekend,
> > Fabian
> >
> 
> >>
> >
>
>