Use Partitioner to forward messages to subtask by index

Is it possible to use a custom Partitioner to forward messages to a downstream substask by the subtask's index? I realize that it would not be safe to rely upon this across job restarts but this does not effect my particular application. I attempted to return a partition number identical to the

Re: Job Manager Configuration

haven’t had to do anything special for the akka configurations. From: Joshua Griffith [mailto:jgriff...@campuslabs.com] Sent: Thursday, November 16, 2017 2:57 PM To: Chan, Regina [Tech] Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Job Manager Configuration I have an

Re: Job Manager Configuration

I have an IO-dominated batch job with 471 distinct tasks (3786 tasks with parallelism) running on 8 nodes with 12 GiB of memory and 4 CPUs each. I haven’t had any problems adding additional tasks except for 1) tasks timing out the first time the cluster is started (I suppose the JVM needs to

Re: Delta iteration not spilling to disk

ab808d51bc1a3b6%7C809fd6c8b87647a9abe28be2888f4a55%7C0%7C0%7C636445440850807259=88tdKjTu8QbisJdVdQdFlJmegNpHPdUSVEOF8EBeNx0%3D=0> 2017-10-25 16:58 GMT+02:00 Joshua Griffith <jgriff...@campuslabs.com<mailto:jgriff...@campuslabs.com>>: Hello Fabian, Thank you for your response. I tried setting the solu

Re: Case Class TypeInformation

23:26 GMT+02:00 Joshua Griffith <jgriff...@campuslabs.com<mailto:jgriff...@campuslabs.com>>: Correction: I have the row’s RowTypeInfo at runtime before the job starts. I don’t have RowTypeInfo at compile time. On Oct 16, 2017, at 4:15 PM, Joshua Griffith <jgriff...@campuslabs.com&l

Re: Delta iteration not spilling to disk

on the heap by setting the solution set to unManaged (DeltaIteration.setSolutionSetUnManaged(true)). Best, Fabian 2017-10-24 21:09 GMT+02:00 Joshua Griffith <jgriff...@campuslabs.com<mailto:jgriff...@campuslabs.com>>: I’m currently using a delta iteration within a batch job and recei

Delta iteration not spilling to disk

I’m currently using a delta iteration within a batch job and received the following error: java.lang.RuntimeException: Memory ran out. Compaction failed. numPartitions: 32 minPartition: 11 maxPartition: 24 number of overflow segments: 0 bucketSize: 125 Overall memory: 23232512 Partition

Re: Case Class TypeInformation

Correction: I have the row’s RowTypeInfo at runtime before the job starts. I don’t have RowTypeInfo at compile time. On Oct 16, 2017, at 4:15 PM, Joshua Griffith <jgriff...@campuslabs.com<mailto:jgriff...@campuslabs.com>> wrote: Hello, I have a case class that wraps a Flink Row

Case Class TypeInformation

Hello, I have a case class that wraps a Flink Row and I’d like to use fields from that Row in a delta iteration join condition. I only have the row’s fields after the job starts. I can construct RowTypeInfo for the Row but I’m not sure how to add that to Flink’s generated type information for

Re: Disable job graph in web UI

Upon further inspection, it appears that the web UI redraws each DOM element with every update. So I think removing the graph won’t fix the page performance issue because each task list item is being redrawn on every refresh. > On Sep 7, 2017, at 2:22 PM, Joshua Griffith <

Disable job graph in web UI

Hello, I have an auto-generated job that creates too many tasks for web UI’s job graph to handle. The browser pinwheels while the page attempts to load. Is it possible to disable the job graph component in the web UI? For slightly smaller jobs, once the graph loads the rest of the UI is usable.

Re: Using Azure Blob Storage with Flink

mqNxdf221hJkcXB%2FMce5GBiMV9KZW1D5EQ%3D=0> : The built jar file, named hadoop-azure.jar, also declares transitive dependencies on the additional artifacts it requires, notably the Azure Storage SDK for Java. On Tue, Aug 29, 2017 at 3:24 PM, Joshua Griffith <jgriff...@campuslabs.com<mailto:jgriff..

Using Azure Blob Storage with Flink

I’m attempting to write to Azure Blob Storage using Flink's FileOutputFormat. I’ve included hadoop-azure within the jar I submit to Flink and configured the paths to be prefixed with

Re: Nested Field Expressions with Rows

cit val rowTpe: TypeInformation[Row] = new RowTypeInfo( >> Array[TypeInformation[_]](BasicTypeInfo.INT_TYPE_INFO, >> BasicTypeInfo.INT_TYPE_INFO), >> Array("id", "value") >> ) >> When you do this, you can also remove move the ResultTypeQueryable interface

Re: Nested Field Expressions with Rows

ava/org/apache/flink/api/common/operators/Keys.java#L329-L331> > On Jul 10, 2017, at 11:54 AM, Joshua Griffith <jgriff...@campuslabs.com> > wrote: > > Hello Fabian, > > Thank you for your response. I tried your recommendation but I’m getting the > same issue. Here’

Re: Nested Field Expressions with Rows

ow] = new RowTypeInfo( > Array[TypeInformation[_]](BasicTypeInfo.INT_TYPE_INFO, > BasicTypeInfo.INT_TYPE_INFO), > Array("id", "value") > ) > When you do this, you can also remove move the ResultTypeQueryable interface > from the MapFunction. > > Cheers

Re: Flink Jobs disappers

Are your containers on separate nodes? Are you running in Kubernetes? Have you set hard resource limits? When I’ve run into this issue it’s been because the JobManager was restarted (I wasn’t running in HA mode). Your node could have been restarted or Docker could have OOM-killed the process

Re: Nested Field Expressions with Rows

e of the query you are trying to run? > Maybe Timo or Fabian (cc'd) can help. > > > Nico > > On Friday, 7 July 2017 23:09:09 CEST Joshua Griffith wrote: >> Hello, >> >> When using nested field expressions like “Account.Id" with nested rows, I >&g

Nested Field Expressions with Rows

Hello, When using nested field expressions like “Account.Id" with nested rows, I get the following error, “This type (GenericType) cannot be used as key.” Is there a way to make nested field expressions work with nested rows? Thanks, Joshua