[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @tillrohrmann Please review ---
[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @dawidwys Thanks for your suggestions. New commit extend `SchemaCoder` with `writeSchema` method that helps to move the writing schema logic away from `AvroSerializationSchema` as you suggested. Totally agree with you that having dynamic `subject` variables make the implementations more generic but as this is `Confluent` specific implementations and this variable is only presented for `ConfluentRegistryAvroSerializationSchema`, I think a user of this method should be aware of how `Confluent` requires this variable when they setup their Kafka Producer and Schema Registry. I am open to suggestions to fix the issue ( by changing `FlinkKafkaProducer`) if you still thinking this is a blocker for this PR. ---
[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @dawidwys in last commit, I did extend `SchemaCoder` to have `getSchemaId` as you suggested. ---
[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @dawidwys For second issue I am looking at other Schema registries and trying to extend `SchemaCoder` ---
[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @dawidwys Thanks! As far as I dog on Confluent code, their api needs `subject` to retrieve the Schema Id and version and it should be provided by consumer. https://github.com/confluentinc/schema-registry/blob/master/client/src/main/java/io/confluent/kafka/schemaregistry/client/SchemaRegistryClient.java#L30 Purpose of new commit is to address your first comments by removing `topic` name in the serialization constructor and replace it with `subject`. So this way serializer doesn't need to know about the `topic` name. If you still see issues with this approach I would appreciate it if you help me to find a better solution. ---
[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @dawidwys I update the PR, please review the usage would be like this ` ConfluentRegistryAvroSerializationSchema.forSpecific(User.class, subject, schemaRegistryUrl)` as Confluent needs "subject" to fetch the Schema info. Now `ConfluentRegistryAvroSerializationSchema` uses "subject" directly instated on `topic + "-value"`. ---
[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @dawidwys Thanks a lot for the review. I will start look into both problems that you've pointed out. Second point would be easy to fix but for the first one I need do some research and see the feasibility of it. ---
[GitHub] flink issue #6269: [FLINK-9770][rest] Fix jar listing
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6269 +1 ---
[GitHub] flink issue #6270: [FLINK-9769][rest] Clear FileUpload attribute after acces...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6270 +1 I rebuilt the 1.5.1 with this changes and now `Upload` jar file from WebUI working. ---
[GitHub] flink issue #6259: [FLINK-9679] Implement AvroSerializationSchema
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6259 @tillrohrmann I implemented `ConfluentRegistryAvroSerializationSchema` in this PR. I would appreciate if you review. ---
[GitHub] flink pull request #6259: [FLINK-9679] Implement AvroSerializationSchema
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6259 [FLINK-9679] Implement AvroSerializationSchema ## What is the purpose of the change Provides implementation of AvroSerializationSchema that write records serialized as avro and also provides version that uses Confluent Schema Registry to write the record. This is following AvroDESerializationSchema implementation patterns to have a consistent code base for Ser/Des. ## Brief change log - Implemented AvroSerializationSchema / RegistryAvroSerializationSchema / ConfluentRegistryAvroSerializationSchema ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (yes) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (JavaDocs) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9679 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6259.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6259 commit 975d1b3a774c11acf01ff2533876d32afbb670b0 Author: Yadan.JS Date: 2018-06-28T16:39:30Z [FLINK-9679] Implement AvroSerializationSchema ---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann PR has been updated! I had to do some changes: - map `User` object to `String` in `TestAvroConsumerConfluent` class - use `read_messages_from_kafka` instead of `read_messages_from_kafka_avro` for vaidation as the events are `String` not `Avro` anymore - removed `kafka-avro` dependency When we add `AvroSerializationConfluentSchema` to Flink, I will update the test as we discussed. ---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann make sense. I created this ticket https://issues.apache.org/jira/browse/FLINK-9679 for implementing `AvroSerializationConfluentSchema`. I will make a PR for it shortly. Mean time I will update this PR also and remove `AvroSerializationConfluentSchema` and using `SimpleStringSchema` which later we can update the test. ---
[GitHub] flink issue #5996: [FLINK-9343] [Example] Add Async Example with External Re...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5996 @StephanEwen I would appreciate it if you do the review after my changes! ---
[GitHub] flink issue #6084: [FLINK-8654][Docs] Extend quickstart docs on how to submi...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6084 @zentol PR has been updated! Please review. ---
[GitHub] flink pull request #5995: [FLINK-9337] Implemented AvroDeserializationSchema
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r198150284
--- Diff:
flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/ConfluentRegistryAvroDeserializationSchema.java
---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.avro.registry.confluent;
+
+import org.apache.flink.formats.avro.AvroDeserializationSchema;
+import org.apache.flink.formats.avro.RegistryAvroDeserializationSchema;
+import org.apache.flink.formats.avro.SchemaCoder;
+
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.specific.SpecificRecord;
+
+import javax.annotation.Nullable;
+
+/**
+ * Deserialization schema that deserializes from Avro binary format using
{@link SchemaCoder} that uses
+ * Confluent Schema Registry.
+ *
+ * @param type of record it produces
+ */
+public class ConfluentRegistryAvroDeserializationSchema extends
RegistryAvroDeserializationSchema {
+
+ private static final int DEFAULT_IDENTITY_MAP_CAPACITY = 1000;
+
+ private static final long serialVersionUID = -1671641202177852775L;
+
+ /**
+* Creates a Avro deserialization schema.
+*
+* @param recordClazz class to which deserialize. Should be
either
+*{@link SpecificRecord} or {@link
GenericRecord}.
+* @param reader reader's Avro schema. Should be provided
if recordClazz is
+*{@link GenericRecord}
+* @param schemaCoderProvider provider for schema coder that reads
writer schema from Confluent Schema Registry
+*/
+ private ConfluentRegistryAvroDeserializationSchema(Class
recordClazz, @Nullable Schema reader,
+ SchemaCoder.SchemaCoderProvider schemaCoderProvider) {
+ super(recordClazz, reader, schemaCoderProvider);
+ }
+
+ /**
+* Creates {@link ConfluentRegistryAvroDeserializationSchema} that
produces {@link GenericRecord}
+* using provided reader schema and looks up writer schema in Confluent
Schema Registry.
+*
+* @param schema schema of produced records
+* @param urlurl of schema registry to connect
+* @return deserialized record in form of {@link GenericRecord}
+*/
+ public static ConfluentRegistryAvroDeserializationSchema
forGeneric(Schema schema, String url) {
+ return forGeneric(schema, url, DEFAULT_IDENTITY_MAP_CAPACITY);
+ }
+
+ /**
+* Creates {@link ConfluentRegistryAvroDeserializationSchema} that
produces {@link GenericRecord}
+* using provided reader schema and looks up writer schema in Confluent
Schema Registry.
+*
+* @param schema schema of produced records
+* @param url url of schema registry to connect
+* @param identityMapCapacity maximum number of cached schema versions
(default: 1000)
+* @return deserialized record in form of {@link GenericRecord}
+*/
+ public static ConfluentRegistryAvroDeserializationSchema
forGeneric(Schema schema, String url,
+ int identityMapCapacity) {
+ return new ConfluentRegistryAvroDeserializationSchema<>(
+ GenericRecord.class,
+ schema,
+ new CachedSchemaCoderProvider(url,
identityMapCapacity));
+ }
+
+ /**
+* Creates {@link AvroDeserializationSchema} that produces classes that
were generated from avro
+* schema and looks up writer schema in Confluent Schema Registry.
+*
+* @param tClass class of record to be produced
+* @param urlurl of schema registry to connect
+* @
[GitHub] flink pull request #5995: [FLINK-9337] Implemented AvroDeserializationSchema
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r198150153
--- Diff:
flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/ConfluentRegistryAvroDeserializationSchema.java
---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.avro.registry.confluent;
+
+import org.apache.flink.formats.avro.AvroDeserializationSchema;
+import org.apache.flink.formats.avro.RegistryAvroDeserializationSchema;
+import org.apache.flink.formats.avro.SchemaCoder;
+
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.specific.SpecificRecord;
+
+import javax.annotation.Nullable;
+
+/**
+ * Deserialization schema that deserializes from Avro binary format using
{@link SchemaCoder} that uses
+ * Confluent Schema Registry.
+ *
+ * @param type of record it produces
+ */
+public class ConfluentRegistryAvroDeserializationSchema extends
RegistryAvroDeserializationSchema {
+
+ private static final int DEFAULT_IDENTITY_MAP_CAPACITY = 1000;
+
+ private static final long serialVersionUID = -1671641202177852775L;
+
+ /**
+* Creates a Avro deserialization schema.
+*
+* @param recordClazz class to which deserialize. Should be
either
+*{@link SpecificRecord} or {@link
GenericRecord}.
+* @param reader reader's Avro schema. Should be provided
if recordClazz is
+*{@link GenericRecord}
+* @param schemaCoderProvider provider for schema coder that reads
writer schema from Confluent Schema Registry
+*/
+ private ConfluentRegistryAvroDeserializationSchema(Class
recordClazz, @Nullable Schema reader,
+ SchemaCoder.SchemaCoderProvider schemaCoderProvider) {
+ super(recordClazz, reader, schemaCoderProvider);
+ }
+
+ /**
+* Creates {@link ConfluentRegistryAvroDeserializationSchema} that
produces {@link GenericRecord}
+* using provided reader schema and looks up writer schema in Confluent
Schema Registry.
+*
+* @param schema schema of produced records
+* @param urlurl of schema registry to connect
+* @return deserialized record in form of {@link GenericRecord}
+*/
+ public static ConfluentRegistryAvroDeserializationSchema
forGeneric(Schema schema, String url) {
+ return forGeneric(schema, url, DEFAULT_IDENTITY_MAP_CAPACITY);
+ }
+
+ /**
+* Creates {@link ConfluentRegistryAvroDeserializationSchema} that
produces {@link GenericRecord}
+* using provided reader schema and looks up writer schema in Confluent
Schema Registry.
+*
+* @param schema schema of produced records
+* @param url url of schema registry to connect
+* @param identityMapCapacity maximum number of cached schema versions
(default: 1000)
+* @return deserialized record in form of {@link GenericRecord}
+*/
+ public static ConfluentRegistryAvroDeserializationSchema
forGeneric(Schema schema, String url,
--- End diff --
@dawidwys couldn't this be `private`?
---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann here is my two cents: By converting the events to `String` and sending them to Kafka there is a high risk to produce a `bad events` and we will lose the benifit of Schema Registry to avoid this malformed events sent to the topic. what you think about `AvroSerializationConfluentSchema` to Flink dist? If we move this serialization code in Flink dist we can drop `kafka-avro` dependencies from here. but still if you think using `String` is ok I can update the PR and use `String` instead of `AvroSerializationConfluentSchema` ---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann I made the changes and used `ConfluentRegistryAvroDeserializationSchema` for the Deserializations. I still using `AvroSerializationConfluentSchema` as we need to compare each income `Event` with `Schema` before sending the data to Kafka and use the schema registry concept to have a full end2end test. We might need to `AvroSerializationConfluentSchema` also to the Flink dist. ---
[GitHub] flink pull request #6083: [FLINK-8983] End-to-end test: Confluent schema reg...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6083#discussion_r197994751
--- Diff: flink-end-to-end-tests/flink-confluent-schema-registry/pom.xml ---
@@ -0,0 +1,228 @@
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.6-SNAPSHOT
+
+ 4.0.0
+
+ flink-confluent-schema-registry
+
+
UTF-8
+ 2.11
+ 4.1.0
+ 0.2.6
+ 1.8
+
+
+
+
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-clients_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-scala_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-streaming-scala_${scala.binary.version}
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-0.10_${scala.binary.version}
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-0.8_${scala.binary.version}
+ ${project.version}
+
+
+
+ org.apache.flink
+ flink-avro
+ ${project.version}
+
+
+
+ org.apache.avro
+ avro
+
+
+ io.confluent
+ kafka-avro-serializer
+ ${confluent.version}
+
--- End diff --
We need this for `AvroSerializationConfluentSchema` to check Events with
Schema before sending data to Kafka
---
[GitHub] flink pull request #6083: [FLINK-8983] End-to-end test: Confluent schema reg...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6083#discussion_r197994770
--- Diff: flink-end-to-end-tests/flink-confluent-schema-registry/pom.xml ---
@@ -0,0 +1,155 @@
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.6-SNAPSHOT
+
+ 4.0.0
+
+ flink-confluent-schema-registry
+
+
UTF-8
+ 3.2.0
+
+
+
+
+ confluent
+ http://packages.confluent.io/maven/
+
+
+
+
+
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-clients_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-streaming-java_${scala.binary.version}
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-0.10_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+ flink-avro
+ ${project.version}
+
+
+ io.confluent
+ kafka-avro-serializer
+ ${confluent.version}
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+ 3.0.0
+
+
+ package
+
+ shade
+
+
+
TestAvroConsumerConfluent
+
+
+
com.google.code.findbugs:jsr305
+
+
+
+
+
*:*
+
+
META-INF/*.SF
+
META-INF/*.DSA
+
META-INF/*.RSA
+
+
+
+
+
+
org.apache.flink.schema.registry.test.TestAvroConsumerConfluent
+
+
+
+
+
+
+
+ org.apache.avro
+ avro-maven-plugin
+ ${avro.version}
+
+
+ generate-sources
+
+ schema
+
+
+
${project.basedir}/src/main/avro/
+
${project.basedir}/
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @dawidwys do u have an example using `ConfluentRegistryAvroDeserializationSchema`. I have some difficulties to make it work ---
[GitHub] flink pull request #6083: [FLINK-8983] End-to-end test: Confluent schema reg...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6083#discussion_r197493383
--- Diff: flink-end-to-end-tests/flink-confluent-schema-registry/pom.xml ---
@@ -0,0 +1,155 @@
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.6-SNAPSHOT
+
+ 4.0.0
+
+ flink-confluent-schema-registry
+
+
UTF-8
+ 3.2.0
+
+
+
+
+ confluent
+ http://packages.confluent.io/maven/
+
+
+
+
+
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-clients_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-streaming-java_${scala.binary.version}
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-0.10_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+ flink-avro
+ ${project.version}
+
+
+ io.confluent
+ kafka-avro-serializer
+ ${confluent.version}
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+ 3.0.0
+
+
+ package
+
+ shade
+
+
+
TestAvroConsumerConfluent
+
+
+
com.google.code.findbugs:jsr305
+
+
+
+
+
*:*
+
+
META-INF/*.SF
+
META-INF/*.DSA
+
META-INF/*.RSA
+
+
+
+
+
+
org.apache.flink.schema.registry.test.TestAvroConsumerConfluent
+
+
+
+
+
+
+
+ org.apache.avro
+ avro-maven-plugin
+ ${avro.version}
+
+
+ generate-sources
+
+ schema
+
+
+
${project.basedir}/src/main/avro/
+
${project.basedir}/
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann PR is updated please review! I will also look into `ConfluentRegistryAvroDeserializationSchema` ---
[GitHub] flink pull request #6083: [FLINK-8983] End-to-end test: Confluent schema reg...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6083#discussion_r197128954
--- Diff:
flink-end-to-end-tests/test-scripts/test-confluent-schema-registry.sh ---
@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/kafka-common.sh
+
+function verify_output {
+ local expected=$(printf $1)
+
+ if [[ "$2" != "$expected" ]]; then
+echo "Output from Flink program does not match expected output."
+echo -e "EXPECTED FOR KEY: --$expected--"
+echo -e "ACTUAL: --$2--"
+PASS=""
+exit 1
+ fi
+}
+
+function test_cleanup {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_kafka_cluster
+ stop_confluent_schema_registry
+
+ # revert our modifications to the Flink distribution
+ mv -f $FLINK_DIR/conf/flink-conf.yaml.bak $FLINK_DIR/conf/flink-conf.yaml
+
+ # make sure to run regular cleanup as well
+ cleanup
+}
+
+trap test_cleanup INT
+trap test_cleanup EXIT
+
+setup_kafka_dist
+setup_confluent_dist
+
+cd flink-end-to-end-tests/flink-confluent-schema-registry
+mvn clean package -Pbuild-jar -nsu
+
+start_kafka_cluster
+start_confluent_schema_registry
+sleep 5
+
+# modify configuration to use port 8082 for Flink
+cp $FLINK_DIR/conf/flink-conf.yaml $FLINK_DIR/conf/flink-conf.yaml.bak
+sed -i -e "s/web.port: 8081/web.port: 8082/"
$FLINK_DIR/conf/flink-conf.yaml
+
+TEST_PROGRAM_JAR=target/flink-confluent-schema-registry-1.6-SNAPSHOT.jar
+
+INPUT_MESSAGE_1='{"name":"Alyssa","favoriteNumber":"250","favoriteColor":"green","eventType":"meeting"}'
+INPUT_MESSAGE_2='{"name":"Charlie","favoriteNumber":"10","favoriteColor":"blue","eventType":"meeting"}'
+INPUT_MESSAGE_3='{"name":"Ben","favoriteNumber":"7","favoriteColor":"red","eventType":"meeting"}'
+USER_SCHEMA='{"namespace":"example.avro","type":"record","name":"User","fields":[{"name":"name","type":"string","default":""},{"name":"favoriteNumber","type":"string","default":""},{"name":"favoriteColor","type":"string","default":""},{"name":"eventType","type":{"name":"EventType","type":"enum","symbols":["meeting"]}}]}'
+
+curl -X POST \
+ http://localhost:8081/subjects/users-value/versions \
+ -H 'cache-control: no-cache' \
+ -H 'content-type: application/vnd.schemaregistry.v1+json' \
+ -d '{"schema": "{\"namespace\": \"example.avro\",\"type\":
\"record\",\"name\": \"User\",\"fields\": [{\"name\": \"name\", \"type\":
\"string\", \"default\": \"\"},{\"name\": \"favoriteNumber\", \"type\":
\"string\", \"default\": \"\"},{\"name\": \"favoriteColor\", \"type\":
\"string\", \"default\": \"\"},{\"name\": \"eventType\",\"type\": {\"name\":
\"EventType\",\"type\": \"enum\", \"symbols\": [\"meeting\"] }}]}"}'
--- End diff --
I tried to remove it but seems Schema Registry needs it and not working
with fist record
---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @dawidwys Thanks for the review! I will look into that PR. This test is for Release 1.4 and 1.5 and provide a proof that Flink V4.0 and V5.0 can work with ConfluentSchema Registry properly. As PR you mentioned will go to next release I think this test still would be useful the way it is and for sure we can update it later. @tillrohrmann Any thoughts? ---
[GitHub] flink issue #6138: [FLINK-9550][DOC]FlinkCEP snippet example has some syntax...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6138 @dawidwys PR's been updated! Please review. ---
[GitHub] flink issue #6138: [FLINK-9550][DOC]FlinkCEP snippet example has some syntax...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6138 @dawidwys Thanks for the review! ---
[GitHub] flink pull request #6138: [FLINK-9550][DOC]FlinkCEP snippet example has some...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6138#discussion_r195749896
--- Diff: docs/dev/libs/cep.md ---
@@ -131,7 +131,7 @@ val result: DataStream[Alert] =
patternStream.select(createAlert(_))
The pattern API allows you to define complex pattern sequences that you
want to extract from your input stream.
Each complex pattern sequence consists of multiple simple patterns, i.e.
patterns looking for individual events with the same properties. From now on,
we will call these simple patterns **patterns**, and the final complex pattern
sequence we are searching for in the stream, the **pattern sequence**. You can
see a pattern sequence as a graph of such patterns, where transitions from one
pattern to the next occur based on user-specified
-*conditions*, e.g. `event.getName().equals("start")`. A **match** is a
sequence of input events which visits all
--- End diff --
just to make it consistent with the `return event.getName().equals("end");`
in the above snippet
---
[GitHub] flink pull request #6138: [FLINK-9550][DOC]FlinkCEP snippet example has some...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6138#discussion_r195749997
--- Diff: docs/dev/libs/cep.md ---
@@ -275,36 +275,38 @@ with "foo", and if the sum of the prices of the
previously accepted events for t
{% highlight java %}
-middle.oneOrMore().where(new IterativeCondition() {
-@Override
-public boolean filter(SubEvent value, Context ctx) throws
Exception {
-if (!value.getName().startsWith("foo")) {
-return false;
-}
-
-double sum = value.getPrice();
-for (Event event : ctx.getEventsForPattern("middle")) {
-sum += event.getPrice();
+middle.oneOrMore()
+.subtype(SubEvent.class)
--- End diff --
will update!
---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann Thanks for the review! I will go through them and will make the changes shortly. ---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @dmpour23 Thanks! I updated the import as`*.util.serialization` was Deprecated! ---
[GitHub] flink issue #6146: [FLINK-9564][tests] Expose flink-end-to-end module direct...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6146 +1 ---
[GitHub] flink issue #6145: [FLINK-9564][tests] Expose flink-end-to-end module direct...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6145 +1 ---
[GitHub] flink issue #6084: [FLINK-8654][Docs] Extend quickstart docs on how to submi...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6084 @zentol sure, will update the PR shortly! ---
[GitHub] flink issue #6139: [FLINK-9551][DOCS]FlinkCEP Scala Combining Patterns table...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6139 @zentol Please review ---
[GitHub] flink issue #6136: [FLINK-4303] [CEP] Add CEP examples
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6136 +1 @kisimple Thank you for update! ---
[GitHub] flink pull request #6136: [FLINK-4303] [CEP] Add CEP examples
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6136#discussion_r194052688
--- Diff:
flink-examples/flink-examples-cep/src/main/java/org/apache/flink/cep/examples/java/monitoring/TemperatureMonitoring.java
---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cep.examples.java.monitoring;
+
+import org.apache.flink.cep.CEP;
+import org.apache.flink.cep.PatternFlatSelectFunction;
+import org.apache.flink.cep.PatternSelectFunction;
+import org.apache.flink.cep.PatternStream;
+import
org.apache.flink.cep.examples.java.monitoring.events.MonitoringEvent;
+import
org.apache.flink.cep.examples.java.monitoring.events.TemperatureAlert;
+import
org.apache.flink.cep.examples.java.monitoring.events.TemperatureEvent;
+import
org.apache.flink.cep.examples.java.monitoring.events.TemperatureWarning;
+import
org.apache.flink.cep.examples.java.monitoring.sources.MonitoringEventSource;
+import org.apache.flink.cep.pattern.Pattern;
+import org.apache.flink.cep.pattern.conditions.SimpleCondition;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.IngestionTimeExtractor;
+import org.apache.flink.streaming.api.windowing.time.Time;
+import org.apache.flink.util.Collector;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CEP example monitoring program.
+ * This example program generates a stream of monitoring events which are
analyzed using
+ * Flink's CEP library. The input event stream consists of temperature and
power events
+ * from a set of racks. The goal is to detect when a rack is about to
overheat.
+ * In order to do that, we create a CEP pattern which generates a
TemperatureWarning
+ * whenever it sees two consecutive temperature events in a given time
interval whose temperatures
+ * are higher than a given threshold value. A warning itself is not
critical but if we see
+ * two warning for the same rack whose temperatures are rising, we want to
generate an alert.
+ * This is achieved by defining another CEP pattern which analyzes the
stream of generated
+ * temperature warnings.
+ */
+public class TemperatureMonitoring {
+
+ private static final double TEMPERATURE_THRESHOLD = 100;
+
+ public static void main(String[] args) throws Exception {
+ System.out.println("Executing temperature monitoring Java
example.");
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+
+ // Use ingestion time => TimeCharacteristic == EventTime +
IngestionTimeExtractor
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+
+ // Input stream of monitoring events
+ DataStream inputEventStream =
env.addSource(new MonitoringEventSource())
+ .assignTimestampsAndWatermarks(new
IngestionTimeExtractor<>());
+
+ // Warning pattern: Two consecutive temperature events whose
temperature is higher
+ // than the given threshold appearing within a time interval of
10 seconds
+ Pattern warningPattern = Pattern
+ .begin("first")
+ .subtype(TemperatureEvent.class)
+ .where(new SimpleCondition() {
+ @Override
+ public boolean filter(TemperatureEvent
event) throws Exception {
+ return event.getTemperature() >
TEMPERATURE_THRESHOLD;
+ }
+ })
+ .next("second")
[GitHub] flink issue #6141: flink-metrics-datadog: beautify metric name by excluding ...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6141 @DmitryBe Thank you! is there a Jira ticket associate to this? I suggest add Jira ticket and update PR title according the Contribution guideline ---
[GitHub] flink pull request #6136: FLINK-4303] [CEP] Add CEP examples
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6136#discussion_r193934789
--- Diff:
flink-examples/flink-examples-cep/src/main/scala/org/apache/flink/cep/examples/scala/monitoring/TemperatureMonitoring.scala
---
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cep.examples.scala.monitoring
+
+import
org.apache.flink.cep.examples.scala.monitoring.events.{MonitoringEvent,
TemperatureAlert, TemperatureEvent, TemperatureWarning}
+import
org.apache.flink.cep.examples.scala.monitoring.sources.MonitoringEventSource
+import org.apache.flink.cep.scala.CEP
+import org.apache.flink.cep.scala.pattern.Pattern
+import org.apache.flink.streaming.api.TimeCharacteristic
+import org.apache.flink.streaming.api.functions.IngestionTimeExtractor
+import org.apache.flink.streaming.api.scala.{DataStream,
StreamExecutionEnvironment, createTypeInformation}
+import org.apache.flink.streaming.api.windowing.time.Time
+
+/**
+ * CEP example monitoring program.
+ * This example program generates a stream of monitoring events which are
analyzed using
+ * Flink's CEP library. The input event stream consists of temperature
and power events
+ * from a set of racks. The goal is to detect when a rack is about to
overheat.
+ * In order to do that, we create a CEP pattern which generates a
TemperatureWarning
+ * whenever it sees two consecutive temperature events in a given time
interval whose temperatures
+ * are higher than a given threshold value. A warning itself is not
critical but if we see
+ * two warning for the same rack whose temperatures are rising, we want
to generate an alert.
+ * This is achieved by defining another CEP pattern which analyzes the
stream of generated
+ * temperature warnings.
+ */
+object TemperatureMonitoring {
+
+ private val TEMPERATURE_THRESHOLD = 100
+
+ def main(args: Array[String]) {
+println("Executing temperature monitoring Scala example.")
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+
+// Use ingestion time => TimeCharacteristic == EventTime +
IngestionTimeExtractor
+env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
+
+// Input stream of monitoring events
+val inputEventStream = env.addSource(new MonitoringEventSource())
+ .assignTimestampsAndWatermarks(new
IngestionTimeExtractor[MonitoringEvent])
+
+// Warning pattern: Two consecutive temperature events whose
temperature is higher
+// than the given threshold appearing within a time interval of 10
seconds
+val warningPattern = Pattern
+ .begin[MonitoringEvent]("first")
+.subtype(classOf[TemperatureEvent])
+.where(_.temperature > TEMPERATURE_THRESHOLD)
+ .next("second")
+.subtype(classOf[TemperatureEvent])
+.where(_.temperature > TEMPERATURE_THRESHOLD)
+ .within(Time.seconds(10))
+
+// Create a pattern stream from our warning pattern
+val tempPatternStream = CEP.pattern(inputEventStream.keyBy(_.rackID),
warningPattern)
+
+// Generate temperature warnings for each matched warning pattern
+val warnings: DataStream[TemperatureWarning] =
tempPatternStream.select( pattern => {
+val first = pattern("first").head.asInstanceOf[TemperatureEvent]
+val second = pattern("second").head.asInstanceOf[TemperatureEvent]
+new TemperatureWarning(first.rackID, (first.temperature +
second.temperature) / 2)
+ }
+)
+
+// Alert pattern: Two consecutive temperature warnings
+// appearing within a time interval of 20 seconds
+val alertPattern = Pattern
+ .begin[TemperatureWarning]("first")
+ .next("second")
+ .within(Time.seconds(20))
+
+// Creat
[GitHub] flink pull request #6136: FLINK-4303] [CEP] Add CEP examples
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6136#discussion_r193934744
--- Diff:
flink-examples/flink-examples-cep/src/main/java/org/apache/flink/cep/examples/java/monitoring/TemperatureMonitoring.java
---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cep.examples.java.monitoring;
+
+import org.apache.flink.cep.CEP;
+import org.apache.flink.cep.PatternFlatSelectFunction;
+import org.apache.flink.cep.PatternSelectFunction;
+import org.apache.flink.cep.PatternStream;
+import
org.apache.flink.cep.examples.java.monitoring.events.MonitoringEvent;
+import
org.apache.flink.cep.examples.java.monitoring.events.TemperatureAlert;
+import
org.apache.flink.cep.examples.java.monitoring.events.TemperatureEvent;
+import
org.apache.flink.cep.examples.java.monitoring.events.TemperatureWarning;
+import
org.apache.flink.cep.examples.java.monitoring.sources.MonitoringEventSource;
+import org.apache.flink.cep.pattern.Pattern;
+import org.apache.flink.cep.pattern.conditions.SimpleCondition;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.IngestionTimeExtractor;
+import org.apache.flink.streaming.api.windowing.time.Time;
+import org.apache.flink.util.Collector;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CEP example monitoring program.
+ * This example program generates a stream of monitoring events which are
analyzed using
+ * Flink's CEP library. The input event stream consists of temperature and
power events
+ * from a set of racks. The goal is to detect when a rack is about to
overheat.
+ * In order to do that, we create a CEP pattern which generates a
TemperatureWarning
+ * whenever it sees two consecutive temperature events in a given time
interval whose temperatures
+ * are higher than a given threshold value. A warning itself is not
critical but if we see
+ * two warning for the same rack whose temperatures are rising, we want to
generate an alert.
+ * This is achieved by defining another CEP pattern which analyzes the
stream of generated
+ * temperature warnings.
+ */
+public class TemperatureMonitoring {
+
+ private static final double TEMPERATURE_THRESHOLD = 100;
+
+ public static void main(String[] args) throws Exception {
+ System.out.println("Executing temperature monitoring Java
example.");
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+
+ // Use ingestion time => TimeCharacteristic == EventTime +
IngestionTimeExtractor
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+
+ // Input stream of monitoring events
+ DataStream inputEventStream =
env.addSource(new MonitoringEventSource())
+ .assignTimestampsAndWatermarks(new
IngestionTimeExtractor<>());
+
+ // Warning pattern: Two consecutive temperature events whose
temperature is higher
+ // than the given threshold appearing within a time interval of
10 seconds
+ Pattern warningPattern = Pattern
+ .begin("first")
+ .subtype(TemperatureEvent.class)
+ .where(new SimpleCondition() {
+ @Override
+ public boolean filter(TemperatureEvent
event) throws Exception {
+ return event.getTemperature() >
TEMPERATURE_THRESHOLD;
+ }
+ })
+ .next("second")
[GitHub] flink issue #6136: FLINK-4303] [CEP] Add CEP examples
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6136 @kisimple Recently I started to work on same example. I ran your code on my local and it did work perfectly. I could see some checkstyle errors that I think travis might pick them up. I have a suggestion for the sake of example scenario. For the `alerts` event you using `LocalTime` which indicate when `alerts` occurred, it would be useful also add `warnings` timestamp to `alerts` event in order to show when racks temperature passed the threshold, something like this on [line](https://github.com/kisimple/flink/blob/5cd3a374b84b2a7aaedb4c4184caded073e19295/flink-examples/flink-examples-cep/src/main/java/org/apache/flink/cep/examples/java/monitoring/TemperatureMonitoring.java#L133) `out.collect(new TemperatureAlert(first.getRackID(), second.getDatetime())); ` of course it needs to update the `TemperatureAlert` model also. ---
[GitHub] flink pull request #6138: [FLINK-9550][DOC]FlinkCEP snippet example has some...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6138 [FLINK-9550][DOC]FlinkCEP snippet example has some syntax errors ## What is the purpose of the change Fixing FlinkCEP snippet code syntax errors and data type mismatches ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9550 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6138.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6138 commit 4a5bc91bc2c326c1a38fd231d24b300b491b75cc Author: Yadan.JS Date: 2018-06-07T21:52:27Z [FLINK-9550][DOC]FlinkCEP snippet example has some syntax errors ---
[GitHub] flink issue #6128: [FLINK-9539][build] Integrate flink-shaded 4.0
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6128 +1 ---
[GitHub] flink pull request #6139: [FLINK-9551][DOCS]FlinkCEP Scala Combining Pattern...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6139 [FLINK-9551][DOCS]FlinkCEP Scala Combining Patterns table has a missing pattern ## What is the purpose of the change in FlinkCEP documentation section related to Combining Patterns Scala Table has a missing patterns compare to Java table `begin(#pattern_sequence)` and also `begin()` pattern has missing `#name` params ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9551 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6139.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6139 commit 83d709f3ce9c73435d8a2c423f79c3d323a74604 Author: Yadan.JS Date: 2018-06-07T22:07:57Z [FLINK-9551][DOCS]FlinkCEP Scala Combining Patterns table has a missing pattern ---
[GitHub] flink pull request #6137: [FLINK-9549][DOC]Fix FlickCEP Docs broken link and...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6137 [FLINK-9549][DOC]Fix FlickCEP Docs broken link and minor style changes ## What is the purpose of the change Fixing FlickCEP broken link and minor style changes ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9549 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6137 commit 2a9b6ff7b4dc4e2ec624ed18ba37b46d82f93e6d Author: Yadan.JS Date: 2018-06-07T21:33:49Z [FLINK-9549][DOC]Fix FlickCEP Docs broken link and minor style changes ---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann Thanks a lot for doing the review! ---
[GitHub] flink issue #6084: [FLINK-8654][Docs] Extend quickstart docs on how to submi...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6084 @zentol yes, that also works. Do you still need me to change it and extend `Next Steps` or we can keep this to provide a bit more info for different ways to submitting jobs. ---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol Thanks for the review. I made the clean up and did some changes to get the ES dependency from flink-quickstart-test/pom.xml ---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6089#discussion_r193581372
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ org.apache.flink
+ flink-end-to-end-tests
+ 1.6-SNAPSHOT
+
+ 4.0.0
+
+ flink-quickstart-test
+ flink-quickstart-test
+ jar
+
+
+ 2.11
+
+
+
+
+ org.apache.flink
+
flink-streaming-java_${scala.binary.version}
+ ${project.version}
+ provided
+
+
+ org.apache.flink
+
flink-streaming-scala_${scala.binary.version}
+ ${project.version}
--- End diff --
Done!
---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6089#discussion_r193581841
--- Diff: flink-end-to-end-tests/test-scripts/test_quickstarts.sh ---
@@ -51,12 +59,12 @@ sed -i -e ''"$(($position + 1))"'i\
${flink.version}\
--- End diff --
I did some code change to get the ES dependency from
flink-quickstart-test/pom.xml and removed the hardcoded values
---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6089#discussion_r193581567 --- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml --- @@ -0,0 +1,98 @@ + + +http://maven.apache.org/POM/4.0.0; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + + org.apache.flink + flink-end-to-end-tests + 1.6-SNAPSHOT + + 4.0.0 + + flink-quickstart-test + flink-quickstart-test + jar + + + 2.11 --- End diff -- Done! ---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6089#discussion_r193581591
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -0,0 +1,98 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ org.apache.flink
+ flink-end-to-end-tests
+ 1.6-SNAPSHOT
+
+ 4.0.0
+
+ flink-quickstart-test
+ flink-quickstart-test
+ jar
+
+
+ 2.11
+
+
+
+
+ org.apache.flink
+
flink-streaming-java_${scala.binary.version}
+ ${project.version}
+ provided
+
+
+ org.apache.flink
+
flink-streaming-scala_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-connector-elasticsearch5_${scala.binary.version}
+ ${project.version}
+
+
+
+
--- End diff --
Removed!
---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol PR has been updated and usage also changed to `test_quickstarts.sh ` ---
[GitHub] flink pull request #6114: [FLINK-9518][Docs] SSL setup Docs config example h...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6114 [FLINK-9518][Docs] SSL setup Docs config example has wrong keys password ## What is the purpose of the change In creating keystores and turststore section password is set to `password` but in setup config section it is `abc123` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9518 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6114 commit 6bbdfd6ebed69572f60bfacbd36192cce9e9a49e Author: Yadan.JS Date: 2018-06-04T11:33:52Z [FLINK-9518][DOCS]SSL setup Docs config example has wrong keys password ---
[GitHub] flink pull request #6113: [FLINK-9517][Docs]Fixing broken links on CLI and U...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6113 [FLINK-9517][Docs]Fixing broken links on CLI and Upgrade Docs ## What is the purpose of the change Fixes broken links for CLI and Upgrade docs. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9517 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6113 commit e6245ad429e18fa609020c6cea1a94a533d69cb0 Author: Yadan.JS Date: 2018-06-04T11:18:24Z [FLINK-9517][Docs]Fixing broken links on CLI and Upgrade Docs ---
[GitHub] flink issue #6112: [FLINK-9508][Docs]General Spell Check on Flink Docs
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6112 @zentol PR is updated! I would appreciate if you review and again so sorry for mixing concerns in a PR. ---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6112#discussion_r192697051 --- Diff: docs/dev/execution_configuration.md --- @@ -45,41 +45,41 @@ The following configuration options are available: (the default is bold) - **`enableClosureCleaner()`** / `disableClosureCleaner()`. The closure cleaner is enabled by default. The closure cleaner removes unneeded references to the surrounding class of anonymous functions inside Flink programs. With the closure cleaner disabled, it might happen that an anonymous user function is referencing the surrounding class, which is usually not Serializable. This will lead to exceptions by the serializer. -- `getParallelism()` / `setParallelism(int parallelism)` Set the default parallelism for the job. +- `getParallelism()` / `setParallelism(int parallelism)`. Set the default parallelism for the job. --- End diff -- Some of the items has period some doesn't. should we remove period for the ones with? ---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6112#discussion_r192693691
--- Diff: docs/ops/upgrading.md ---
@@ -172,7 +172,7 @@ First major step in job migration is taking a savepoint
of your job running in t
You can do this with the command:
{% highlight shell %}
-$ bin/flink savepoint :jobId [:targetDirectory]
+$ ./bin/flink savepoint [savepointDirectory]
--- End diff --
will revert back!
I was looking at same command on CLI page as uses `savepointDirectory`
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/cli.html#trigger-a-savepoint
but I see `targetDirectory` has also been used for Savepoint documentarians
---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6112#discussion_r192693807 --- Diff: docs/ops/filesystems.md --- @@ -112,10 +111,9 @@ To prevent inactive streams from taking up the complete pool (preventing new con `fs..limit.stream-timeout`. If a stream does not read/write any bytes for at least that amount of time, it is forcibly closed. These limits are enforced per TaskManager, so each TaskManager in a Flink application or cluster will open up to that number of connections. -In addition, the The limit are also enforced only per FileSystem instance. Because File Systems are created per scheme and authority, different +In addition, the limit are also enforced only per FileSystem instance. Because File Systems are created per scheme and authority, different --- End diff -- will change ---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6112#discussion_r192690554
--- Diff: docs/ops/upgrading.md ---
@@ -183,15 +183,15 @@ In this step, we update the framework version of the
cluster. What this basicall
the Flink installation with the new version. This step can depend on how
you are running Flink in your cluster (e.g.
standalone, on Mesos, ...).
-If you are unfamiliar with installing Flink in your cluster, please read
the [deployment and cluster setup documentation]({{ site.baseurl
}}/ops/deployment/cluster_setup.html).
+If you are unfamiliar with installing Flink in your cluster, please read
the [clusters and deployment setup documentation]({{ site.baseurl
}}/ops/deployment/cluster_setup.html).
--- End diff --
will revert back! I waned to make it similar to the `Clusters and
Deployment` page title
---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6112#discussion_r192689975 --- Diff: docs/ops/security-ssl.md --- @@ -22,22 +22,22 @@ specific language governing permissions and limitations under the License. --> -This page provides instructions on how to enable SSL for the network communication between different flink components. +This page provides instructions on how to enable SSL for the network communication between different Flink components. ## SSL Configuration -SSL can be enabled for all network communication between flink components. SSL keystores and truststore has to be deployed on each flink node and configured (conf/flink-conf.yaml) using keys in the security.ssl.* namespace (Please see the [configuration page](config.html) for details). SSL can be selectively enabled/disabled for different transports using the following flags. These flags are only applicable when security.ssl.enabled is set to true. +SSL can be enabled for all network communication between Flink components. SSL keystores and truststore has to be deployed on each Flink node and configured (conf/flink-conf.yaml) using keys in the `security.ssl.*` namespace (Please see the [configuration page](config.html) for details). SSL can be selectively enabled/disabled for different transports using the following flags. These flags are only applicable when `security.ssl.enabled` is set to true. * **taskmanager.data.ssl.enabled**: SSL flag for data communication between task managers * **blob.service.ssl.enabled**: SSL flag for blob service client/server communication -* **akka.ssl.enabled**: SSL flag for the akka based control connection between the flink client, jobmanager and taskmanager -* **jobmanager.web.ssl.enabled**: Flag to enable https access to the jobmanager's web frontend +* **akka.ssl.enabled**: SSL flag for akka based control connection between the Flink client, JobManager and TaskManager --- End diff -- will revert back! I was reading other pages and seems there is some inconsistency on other pages as they used `JobManager` and `TaskManager` fromt ---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6112#discussion_r192689067 --- Diff: docs/ops/filesystems.md --- @@ -70,21 +70,20 @@ That way, Flink seamlessly supports all of Hadoop file systems, and all Hadoop-c - **har** - ... - ## Common File System configurations The following configuration settings exist across different file systems Default File System -If paths to files do not explicitly specify a file system scheme (and authority), a default scheme (and authority) will be used. +If path to files do not explicitly specify a file system scheme (and authority), a default scheme (and authority) will be used. --- End diff -- will revert back ---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6112#discussion_r192689122
--- Diff: docs/ops/filesystems.md ---
@@ -70,21 +70,20 @@ That way, Flink seamlessly supports all of Hadoop file
systems, and all Hadoop-c
- **har**
- ...
-
## Common File System configurations
The following configuration settings exist across different file systems
Default File System
-If paths to files do not explicitly specify a file system scheme (and
authority), a default scheme (and authority) will be used.
+If path to files do not explicitly specify a file system scheme (and
authority), a default scheme (and authority) will be used.
{% highlight yaml %}
fs.default-scheme:
{% endhighlight %}
-For example, if the default file system configured as `fs.default-scheme:
hdfs://localhost:9000/`, then a a file path of
-`/user/hugo/in.txt'` is interpreted as
`hdfs://localhost:9000/user/hugo/in.txt'`
+For example, if the default file system configured as `fs.default-scheme:
hdfs://localhost:9000/`, then a file path of
+`'/user/hugo/in.txt'` is interpreted as
`'hdfs://localhost:9000/user/hugo/in.txt'`
--- End diff --
will do
---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6112#discussion_r192689028 --- Diff: docs/internals/ide_setup.md --- @@ -89,7 +89,7 @@ IntelliJ supports checkstyle within the IDE using the Checkstyle-IDEA plugin. 3. Set the "Scan Scope" to "Only Java sources (including tests)". 4. Select _8.4_ in the "Checkstyle Version" dropdown and click apply. **This step is important, don't skip it!** -5. In the "Configuration File" pane, add a new configuration using the plus icon: +5. In the "Configuration File" page, add a new configuration using the plus icon: --- End diff -- will revert back! ---
[GitHub] flink issue #6112: [FLINK-9508][Docs]General Spell Check on Flink Docs
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6112 @zentol Thanks a lot for review and I am so sorry for causing pain. I will reduce the scope of this PR and only keep spelling issues for now. ---
[GitHub] flink pull request #6112: [FLINK-9508][Docs]General Spell Check on Flink Doc...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6112 [FLINK-9508][Docs]General Spell Check on Flink Docs ## What is the purpose of the change General spell check for Flink docs ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9508 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6112 commit 3695a59f91352eb83b0dddc9f8ff8b54b6e98a32 Author: Yadan.JS Date: 2018-05-29T03:13:59Z [FLINK-9508][Docs]General Spell Check on Flink Docs ---
[GitHub] flink issue #5996: [FLINK-9343] [Example] Add Async Example with External Re...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5996 @StephanEwen PR is updated as requested! I would appreciate if you review. ---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann I would appreciate if you review or assign a reviewer to this PR. ---
[GitHub] flink issue #6084: [FLINK-8654][Docs] Extend quickstart docs on how to submi...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6084 @zentol Please review ---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol PR is updated with requested changes! Please review ---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol PR is updated! ---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6089#discussion_r191740715
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -75,6 +76,8 @@ function verify_result {
}
function shutdown_elasticsearch_cluster {
+ local index=$1
--- End diff --
+1
---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6089#discussion_r191740903 --- Diff: flink-end-to-end-tests/test-scripts/test_quickstarts.sh --- @@ -18,29 +18,38 @@ # End to end test for quick starts test. +# Usage: +# FLINK_DIR= flink-end-to-end-tests/test-scripts/test_quickstarts.sh --- End diff -- will change the name ---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6089#discussion_r191740667
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -56,13 +56,14 @@ function verify_elasticsearch_process_exist {
function verify_result {
local numRecords=$1
+local index=$2
--- End diff --
+1
---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol Thanks! found them :) ---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol I reverted back flink-elasticsearch* modules. Do you think we still need them as they've never been used? ---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request: https://github.com/apache/flink/pull/6089#discussion_r191733069 --- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml --- @@ -86,6 +102,21 @@ under the License. + + org.apache.maven.plugins + maven-enforcer-plugin + + + dependency-convergence + + enforce + + + true --- End diff -- @zentol good point. this is not necessary anymore ---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on a diff in the pull request:
https://github.com/apache/flink/pull/6089#discussion_r191732783
--- Diff: flink-end-to-end-tests/flink-quickstart-test/pom.xml ---
@@ -20,27 +20,43 @@ under the License.
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-
- 4.0.0
-
org.apache.flink
flink-end-to-end-tests
1.6-SNAPSHOT
- ..
+ 4.0.0
- flink-elasticsearch5-test
- flink-elasticsearch5-test
+ flink-quickstart-test
+ flink-quickstart-test
jar
+
+ 2.11
+
+
org.apache.flink
flink-streaming-java_${scala.binary.version}
${project.version}
provided
+
+ org.apache.flink
+
flink-streaming-scala_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-connector-elasticsearch_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-connector-elasticsearch2_${scala.binary.version}
--- End diff --
@zentol removed multiple connector versions
---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol updated the PR as suggested! Please review ---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol sure! I will update the PR with your requested changes. ---
[GitHub] flink issue #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6089 @zentol Thanks for review! Yes, I totally agree as the most of test script is duplicated, we need to refactor them the way you mentioned, I will update the PR with new changes. QQ: One reason I borrowed an example from 'Flink-example' module was to e2e test the actual example code also. One benefit would be testing the actual example and safeguard the 'Flink-example' module from any code changes without passing the tests. (or another cleaner option is to increase unittest coverage to flink-example package). Do you think we should have a dedicated module under test folder or extend the 'Flink-example' module with ES example and test e2e for that. ---
[GitHub] flink pull request #6089: [FLINK-9451]End-to-end test: Scala Quickstarts
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6089 [FLINK-9451]End-to-end test: Scala Quickstarts ## What is the purpose of the change Added an end-to-end test which verifies Flink's quickstarts scala. It does the following: - create a new Flink project using the quickstarts scala archetype - add a new Flink kafka connector dependency to the pom.xml - run mvn clean package - verify that no core dependencies are contained in the jar file - Run the program ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9451 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6089.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6089 commit 26f5948eff55fd54dbffc3125e0599479bee3bbe Author: Yadan.JS <y_shirvany@...> Date: 2018-05-28T16:16:40Z [FLINK-9451]End-to-end test: Scala Quickstarts ---
[GitHub] flink issue #5996: [FLINK-9343] [Example] Add Async Example with External Re...
Github user medcv commented on the issue:
https://github.com/apache/flink/pull/5996
@StephanEwen I updated the code and used `ExecutionContext.global` which
makes use of threadpool.
I tested it and results showed that the code sent off multiple requests
concurrently (parallelism = 1).
Bellow result shows that waiting time was overlaying with sending other
requests and receiving responses
Output format: `(Quote Number, {Quote API response})` which `Quote Number`
was sequential input.
https://user-images.githubusercontent.com/9163132/40592122-6ea8ef5a-61e9-11e8-99d5-2deeffa3d9be.png;>
Bellow result is when I called the external api within streaming API which
shows the call is blocked until request got resolved
https://user-images.githubusercontent.com/9163132/40592123-70b03d62-61e9-11e8-9068-eef2a102f19b.png;>
---
[GitHub] flink issue #6044: [FLINK-1044] Website: Offer a zip archive with a pre-setu...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6044 @zentol Ok, I created a Jira ticket for Scala Quickstarts end2end test and will make a PR for it. ---
[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann please review ---
[GitHub] flink issue #6084: [FLINK-8654][Docs] Extend quickstart docs on how to submi...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6084 @zentol Please review ---
[GitHub] flink pull request #6084: [FLINK-8654][Docs] Extend quickstart docs on how t...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6084 [FLINK-8654][Docs] Extend quickstart docs on how to submit jobs ## Brief change log The quickstart documentation explains how to setup the project, build the jar and run things in the IDE, but neither explains how to submit the jar to a cluster nor guides the user to where he could find this information (like the CLI docs). ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-8654 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6084.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6084 commit 6e8b7434df51e4265a00534de12b6c509d8977fe Author: Yadan.JS <y_shirvany@...> Date: 2018-05-26T20:57:59Z [FLINK-8654][Docs]Extend quickstart docs on how to submit jobs ---
[GitHub] flink pull request #6044: [FLINK-1044] Website: Offer a zip archive with a p...
Github user medcv closed the pull request at: https://github.com/apache/flink/pull/6044 ---
[GitHub] flink issue #5996: [FLINK-9343] [Example] Add Async Example with External Re...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5996 @StephanEwen Thanks for review. make sense, I will update the client code to use Threadpool and will run more tests. ---
[GitHub] flink pull request #6083: [FLINK-8983] End-to-end test: Confluent schema reg...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6083 [FLINK-8983] End-to-end test: Confluent schema registry ## Brief change log Added an end-to-end test which verifies that Flink is able to work together with the Confluent schema registry. In order to do that, this test sets up a Kafka cluster and a Flink job which writes and reads from the Confluent schema registry producing an Avro type. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-8983 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6083.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6083 commit 8bcaee1a6d8b32e10888e46e608a1478b4a66e9b Author: Yadan.JS <y_shirvany@...> Date: 2018-05-21T02:31:26Z [FLINK-8983] End-to-end test: Confluent schema registry ---
[GitHub] flink issue #6044: [FLINK-1044] Website: Offer a zip archive with a pre-setu...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6044 @zentol Thanks for the review. Make sense to build this `quickstarts.zip` package on Jenkins but the main concern was test the `.zip` package (included all dependencies) before uploading it on site. As end2end test for quickstarts already got merged then it should be fine. I will close this PR. Quick side note: I looked at the end2end test for quickstarts (`test-quickstarts.sh`) and it has a test for Java version. Do you think it make sense we add a ent2end test for Scala version of quickstarts also? ---
[GitHub] flink issue #5996: [FLINK-9343] [Example] Add Async Example with External Re...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5996 @zentol I would appreciate if you review this. We use this type of example for enriching our Steam data via API call which is very common use-case and I think it would be useful for other people to have an example for it. ---
[GitHub] flink issue #6044: [FLINK-1044] Website: Offer a zip archive with a pre-setu...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/6044 @StephanEwen I would appreciate if you review this ---
[GitHub] flink pull request #6044: [FLINK-1044] Website: Offer a zip archive with a p...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6044 [FLINK-1044] Website: Offer a zip archive with a pre-setup user project ## What is the purpose of the change This PR will run two tests to build Java and Scala quickstart packages. After compiling and packaging the jar files, it submit the wordcount class to a cluster and validate the output. At the very end it should upload the jar files to `https://flink.apache.org/q` server. Last part not working as I need more details to find out how to upload the .jar files to the flink server and I am looking for help from reviewers to point me to the right direction. ## Brief change log - *add two new e2e tests * - *Modify run-pre-commit-tests* ## Verifying this change This change added tests and can be verified as follows: Run test_build_quickstart_java.sh and test_build_quickstart_scala.sh to verify this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-1044 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6044.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6044 commit dd59a8df1662bd1fa06dbfa220802e20a12945c3 Author: Yadan.JS <y_shirvany@...> Date: 2018-05-18T22:11:40Z [FLINK-1044] Website: Offer a zip archive with a pre-setup user project ---
[GitHub] flink issue #5996: [FLINK-9343] [Example] Add Async Example with External Re...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5996 @tzulitai I would appreciate if you review this. ---
[GitHub] flink pull request #6018: [FLINK-9372] Typo on Elasticsearch website link (e...
GitHub user medcv opened a pull request: https://github.com/apache/flink/pull/6018 [FLINK-9372] Typo on Elasticsearch website link (elastic.io --> elastic.co) ## What is the purpose of the change Typo on Elasticsearch website link elastic.io --> elastic.co ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/medcv/flink FLINK-9372 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6018.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6018 commit 4e42304dce4bfbcab964548e4f153dc8c212b569 Author: Yadan.JS <y_shirvany@...> Date: 2018-05-15T22:03:22Z [FLINK-9372] Typo on Elasticsearch website link ---
[GitHub] flink issue #5823: [FLINK-9008] [e2e] Implements quickstarts end to end test
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5823 @tzulitai I agree it should be separate of this PR. Main challenge is to upload a compiled Quickstart zip file contain all dependencies for each changes (minor) we do automatically. I was thinking after this PR got merged we can re-use this e2e test to build a Quickstart .zip file ready to upload on each merged PR. ---
[GitHub] flink issue #5823: [FLINK-9008] [e2e] Implements quickstarts end to end test
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5823 @zhangminglei @zentol I was looking at the ticket [FLINK-1044](https://issues.apache.org/jira/browse/FLINK-1044) and seems we can use the output of this test to offer a zip archive with a pre-setup user project, upload it to `https://flink.apache.org/q/quickstart.zip` and update the document to point to this .zip file. This can help for users that have no experience with Maven/SBT setups. ---
[GitHub] flink issue #5989: [FLINK-9333] [Docs] QuickStart Docs Spelling fix and some...
Github user medcv commented on the issue: https://github.com/apache/flink/pull/5989 @zentol please review ---
