Hi all.
I have streaming job that reads from Kafka 0.10 manipulates data and write to
Cassandra (Tuple18)
Also this job has window and CustomReducer class involved to reduce data.
If groupedBy_windowed_stream DataStream defined with 9 fields (Tuple9)
compilation takes 5 seconds.
In current (Tuple11) setup it takes more then 16 minutes.
The artifact is Ok, it runs and does the job…
The problem is only to long compilation time.
Has someone been met with this issue?
DataStream<Tuple11<String, String, String, Integer, Integer, Integer, Integer,
String,Integer, Integer,Integer>> forAggregations
= streamParsedWithTimestamps // This stream is Tuple18
.map(x -> new Tuple11<>(x.f6, x.f7, x.f8, x.f0, x.f1, x.f2, x.f3,
x.f11,x.f16,x.f17,new Integer(1)));
WindowedStream<Tuple11<String, String, String, Integer, Integer, Integer,
Integer, String, Integer,Integer,Integer>, Tuple, TimeWindow> windowed_stream =
forAggregations
.keyBy(7)
.window(TumblingProcessingTimeWindows.of(Time.hours(1)));
DataStream<Tuple11<String, String, String, Integer, Integer, Integer, Integer,
String, Integer,Integer,Integer>> groupedBy_windowed_stream = windowed_stream
.reduce(new CustomReducer());
…….
private static class CustomReducer
implements ReduceFunction<Tuple11<String, String, String, Integer,
Integer, Integer, Integer, String, Integer,Integer,Integer>> {
@Override
public Tuple11<String, String, String, Integer, Integer, Integer, Integer,
String, Integer,Integer,Integer> reduce(
Tuple11<String, String, String, Integer, Integer, Integer, Integer,
String, Integer,Integer,Integer> v1,
Tuple11<String, String, String, Integer, Integer, Integer, Integer,
String, Integer,Integer,Integer> v2) throws Exception {
v1.f8 += v2.f8;
v1.f9 += v2.f9;
v1.f10 += v2.f10;
return new
Tuple11<>(v1.f0,v1.f1,v1.f2,v1.f3,v1.f4,v1.f5,v1.f6,v1.f7,v1.f8,v1.f9,v1.f10);
}
}
mvn clean package -Pbuild-jar
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building analytics 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ analytics ---
[INFO] Deleting /Users/gm5806/activity-monitoring/analytics/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ analytics
---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ analytics ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 12 source files to
/Users/gm5806/activity-monitoring/analytics/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
analytics ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/Users/gm5806/activity-monitoring/analytics/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @
analytics ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ analytics ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ analytics ---
[INFO] Building jar:
/Users/gm5806/activity-monitoring/analytics/target/analytics-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-shade-plugin:2.4.1:shade (default) @ analytics ---
[INFO] Including org.slf4j:slf4j-api:jar:1.7.7 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.0.5 in the shaded jar.
[INFO] Including org.apache.flink:force-shading:jar:1.3.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-table_2.11:jar:1.3.2 in the shaded jar.
[INFO] Including org.scala-lang:scala-library:jar:2.11.7 in the shaded jar.
[INFO] Including org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4
in the shaded jar.
[INFO] Including org.apache.flink:flink-cep_2.11:jar:1.3.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-connector-kafka-0.10_2.11:jar:1.3.2 in
the shaded jar.
[INFO] Including org.apache.flink:flink-connector-kafka-0.9_2.11:jar:1.3.2 in
the shaded jar.
[INFO] Including org.apache.flink:flink-connector-kafka-base_2.11:jar:1.3.2 in
the shaded jar.
[INFO] Including org.apache.kafka:kafka-clients:jar:0.10.2.1 in the shaded jar.
[INFO] Including net.jpountz.lz4:lz4:jar:1.3.0 in the shaded jar.
[INFO] Including org.apache.flink:flink-connector-cassandra_2.11:jar:1.3.2 in
the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.0.33.Final in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.0.33.Final in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.0.33.Final in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.0.33.Final in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.0.33.Final in the shaded jar.
[INFO] Including io.dropwizard.metrics:metrics-core:jar:3.1.2 in the shaded jar.
[INFO] Including net.sourceforge.javacsv:javacsv:jar:2.0 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.2.6 in the shaded
jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.2.5 in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.1.1 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.6 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.2.2 in the shaded jar.
[INFO] Including com.typesafe.akka:akka-actor_2.11:jar:2.5.0 in the shaded jar.
[INFO] Including com.typesafe:config:jar:1.3.1 in the shaded jar.
[INFO] Including org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0 in
the shaded jar.
[INFO] Including com.typesafe.akka:akka-http_2.11:jar:10.1.0-RC1 in the shaded
jar.
[INFO] Including com.typesafe.akka:akka-http-core_2.11:jar:10.0.1 in the shaded
jar.
[INFO] Including com.typesafe.akka:akka-parsing_2.11:jar:10.0.1 in the shaded
jar.
[INFO] Including com.typesafe.akka:akka-stream_2.11:jar:2.4.16 in the shaded
jar.
[INFO] Including org.reactivestreams:reactive-streams:jar:1.0.0 in the shaded
jar.
[INFO] Including com.typesafe:ssl-config-core_2.11:jar:0.2.1 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing
/Users/gm5806/activity-monitoring/analytics/target/analytics-1.0-SNAPSHOT.jar
with
/Users/gm5806/activity-monitoring/analytics/target/analytics-1.0-SNAPSHOT-shaded.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:33 min
[INFO] Finished at: 2018-02-18T16:34:27+02:00
[INFO] Final Memory: 48M/503M
[INFO] ------------------------------------------------------------------------
