[GitHub] [flink] flinkbot edited a comment on issue #8303: [FLINK-12343] [flink-yarn] add file replication config for yarn configuration
flinkbot edited a comment on issue #8303: [FLINK-12343] [flink-yarn] add file replication config for yarn configuration URL: https://github.com/apache/flink/pull/8303#issuecomment-511684151 ## CI report: * 6a7ca58b4a04f6dce250045e021702e67e82b893 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119421914) * 4d38a8df0d59734c4b2386689a2f17b9f2b44b12 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119441376) * 9c14836f8639e98d58cf7bb32e38b938b3843994 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119577044) * 76186776c5620598a19234245bbd05dfdfb1c62c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/120113740) * 628ca7b316ad3968c90192a47a84dd01f26e2578 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122381349) * d204a725ff3c8a046cbd1b84e34d9e3ae8aafeac : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123620485) * 143efadbdb6c4681569d5b412a175edfb1633b85 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123637809) * b78b64a82ed2a9a92886095ec42f06d5082ad830 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123671219) * 5145a0b9d6b320456bb971d96b9cc47707c8fd28 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125476639) * 0d4d944c28c59ca1caa6c453c347ec786b40d245 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125762588) * 91552c3804f5e96cc573e6ed48756f2b54c037d4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125844084) * fbf6d2850b6aa7c303981e6f5b24b0da0956b820 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126086321) * 45282a2139ec3c7196ba10b6fb829b0ec275 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126350641) * cc515180d99fa35b482c61c2e264e77760e55dc5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126354591) * b8341665cb9b8bf3559d93ea74715ddac3ca67b2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127617990) * 462a8cf7286f63afcfb5cecbce37df232d3f8ec4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127623432) * 51397b3c71d8dd4549eba379b2b873d2e523e268 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127625943) * 60e035ab71050b367fcf5411953a75a8a3b32a75 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127672092) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-7151) Add a basic function SQL DDL
[ https://issues.apache.org/jira/browse/FLINK-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenqiu Huang updated FLINK-7151: - Description: Based on create function and table.we can register a udf,udaf,udtf use sql: {code} CREATE FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS class_name; DROP FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name; ALTER FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name RENAME TO new_name; {code} {code} CREATE function 'TOPK' AS 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF'; INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP BY id; {code} This ticket can assume that the function class is already loaded in classpath by users. Advanced syntax like to how to dynamically load udf libraries from external locations can be on a separate ticket. was: Based on create function and table.we can register a udf,udaf,udtf use sql: {code} CREATE FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS class_name [WITH (properties.)]; DROP FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name; ALTER FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name RENAME TO new_name; {code} {code} CREATE function 'TOPK' AS 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF'; INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP BY id; {code} This ticket can assume that the function class is already loaded in classpath by users. Advanced syntax like to how to dynamically load udf libraries from external locations can be on a separate ticket. > Add a basic function SQL DDL > > > Key: FLINK-7151 > URL: https://issues.apache.org/jira/browse/FLINK-7151 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: yuemeng >Assignee: Zhenqiu Huang >Priority: Critical > Fix For: 1.10.0 > > > Based on create function and table.we can register a udf,udaf,udtf use sql: > {code} > CREATE FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS > class_name; > DROP FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name; > ALTER FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name RENAME TO > new_name; > {code} > {code} > CREATE function 'TOPK' AS > 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF'; > INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP > BY id; > {code} > This ticket can assume that the function class is already loaded in classpath > by users. Advanced syntax like to how to dynamically load udf libraries from > external locations can be on a separate ticket. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #8303: [FLINK-12343] [flink-yarn] add file replication config for yarn configuration
flinkbot edited a comment on issue #8303: [FLINK-12343] [flink-yarn] add file replication config for yarn configuration URL: https://github.com/apache/flink/pull/8303#issuecomment-511684151 ## CI report: * 6a7ca58b4a04f6dce250045e021702e67e82b893 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119421914) * 4d38a8df0d59734c4b2386689a2f17b9f2b44b12 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119441376) * 9c14836f8639e98d58cf7bb32e38b938b3843994 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119577044) * 76186776c5620598a19234245bbd05dfdfb1c62c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/120113740) * 628ca7b316ad3968c90192a47a84dd01f26e2578 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122381349) * d204a725ff3c8a046cbd1b84e34d9e3ae8aafeac : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123620485) * 143efadbdb6c4681569d5b412a175edfb1633b85 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123637809) * b78b64a82ed2a9a92886095ec42f06d5082ad830 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123671219) * 5145a0b9d6b320456bb971d96b9cc47707c8fd28 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125476639) * 0d4d944c28c59ca1caa6c453c347ec786b40d245 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125762588) * 91552c3804f5e96cc573e6ed48756f2b54c037d4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125844084) * fbf6d2850b6aa7c303981e6f5b24b0da0956b820 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126086321) * 45282a2139ec3c7196ba10b6fb829b0ec275 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126350641) * cc515180d99fa35b482c61c2e264e77760e55dc5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126354591) * b8341665cb9b8bf3559d93ea74715ddac3ca67b2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127617990) * 462a8cf7286f63afcfb5cecbce37df232d3f8ec4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127623432) * 51397b3c71d8dd4549eba379b2b873d2e523e268 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127625943) * 60e035ab71050b367fcf5411953a75a8a3b32a75 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/127672092) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8303: [FLINK-12343] [flink-yarn] add file replication config for yarn configuration
flinkbot edited a comment on issue #8303: [FLINK-12343] [flink-yarn] add file replication config for yarn configuration URL: https://github.com/apache/flink/pull/8303#issuecomment-511684151 ## CI report: * 6a7ca58b4a04f6dce250045e021702e67e82b893 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119421914) * 4d38a8df0d59734c4b2386689a2f17b9f2b44b12 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119441376) * 9c14836f8639e98d58cf7bb32e38b938b3843994 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/119577044) * 76186776c5620598a19234245bbd05dfdfb1c62c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/120113740) * 628ca7b316ad3968c90192a47a84dd01f26e2578 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122381349) * d204a725ff3c8a046cbd1b84e34d9e3ae8aafeac : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123620485) * 143efadbdb6c4681569d5b412a175edfb1633b85 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123637809) * b78b64a82ed2a9a92886095ec42f06d5082ad830 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123671219) * 5145a0b9d6b320456bb971d96b9cc47707c8fd28 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125476639) * 0d4d944c28c59ca1caa6c453c347ec786b40d245 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125762588) * 91552c3804f5e96cc573e6ed48756f2b54c037d4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125844084) * fbf6d2850b6aa7c303981e6f5b24b0da0956b820 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126086321) * 45282a2139ec3c7196ba10b6fb829b0ec275 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126350641) * cc515180d99fa35b482c61c2e264e77760e55dc5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/126354591) * b8341665cb9b8bf3559d93ea74715ddac3ca67b2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127617990) * 462a8cf7286f63afcfb5cecbce37df232d3f8ec4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127623432) * 51397b3c71d8dd4549eba379b2b873d2e523e268 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127625943) * 60e035ab71050b367fcf5411953a75a8a3b32a75 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-14077) get java.util.ConcurrentModificationException when push metrics to PushGateway
zhangjun created FLINK-14077: Summary: get java.util.ConcurrentModificationException when push metrics to PushGateway Key: FLINK-14077 URL: https://issues.apache.org/jira/browse/FLINK-14077 Project: Flink Issue Type: Bug Components: Runtime / Metrics Affects Versions: 1.9.0 Reporter: zhangjun When my flink program is running for a while, I get the following error message {code:java} 2019-09-15 10:11:28,058 WARN org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter - Failed to push metrics to PushGateway with jobName flinkjob_bb51bc6919b89a3e7d278dd0ef1d.java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719) at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)at java.util.AbstractCollection.addAll(AbstractCollection.java:343)at java.util.HashSet.(HashSet.java:119)at org.apache.kafka.common.internals.PartitionStates.partitionSet(PartitionStates.java:66) at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedPartitions(SubscriptionState.java:293) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$ConsumerCoordinatorMetrics$1.measure(ConsumerCoordinator.java:884) at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:61) at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:52) at org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricWrapper.getValue(KafkaMetricWrapper.java:37) at org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricWrapper.getValue(KafkaMetricWrapper.java:27) at org.apache.flink.metrics.prometheus.AbstractPrometheusReporter$2.get(AbstractPrometheusReporter.java:224) at org.apache.flink.shaded.io.prometheus.client.Gauge.collect(Gauge.java:295) at org.apache.flink.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183) at org.apache.flink.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216) at org.apache.flink.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137) at org.apache.flink.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:290) at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105) at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76) at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:436) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} my flink job is a stream job, write to hbase from kafka stream,my kafka version is 0.10,the flink version is 1.9.0 ,the metrics conf is : {code:java} metrics.reporters: promgateway metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter metrics.reporter.promgateway.host: metrics.reporter.promgateway.port: 9091 metrics.reporter.promgateway.jobName: flinkjob_ metrics.reporter.promgateway.randomJobNameSuffix: true metrics.reporter.promgateway.deleteOnShutdown: true {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al…
walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al… URL: https://github.com/apache/flink/pull/9300#discussion_r324441186 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/mapper/MapOpInterface.java ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.mapper; + +import org.apache.flink.types.Row; + +/** + * Interface for the map operation of Row type data. + */ +public interface MapOpInterface { Review comment: should this file be named just `MapOp`. There's already a `MatVecOp` class thought we would like to keep the naming conversion as well? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al…
walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al… URL: https://github.com/apache/flink/pull/9300#discussion_r324441066 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/mapper/FlatModelMapper.java ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.mapper; + +import org.apache.flink.ml.api.misc.param.Params; +import org.apache.flink.table.api.TableSchema; +import org.apache.flink.types.Row; + +import java.util.List; + +/** + * Abstract class for flatMappers with model. + * FlatModelMapper transform one Row type data into zero, one, or more Row type result data. + * Operations that produce multiple strictly one Row type result data per Row type data + * can also use the {@link ModelMapper}. + */ +public abstract class FlatModelMapper extends FlatMapper { + + /** +* schema of the model with Table type. +*/ + protected TableSchema modelSchema; + + public FlatModelMapper(TableSchema modelSchema, TableSchema dataSchema, Params params) { + super(dataSchema, params); + this.modelSchema = modelSchema; + } + + /** +* Load model from the list of Row type data. +* +* @param modelRows the list of Row type data +*/ + public abstract void loadModel(List modelRows); + + /** +* Generate new instance of given FlatModelMapper class without model data. +* The instance can not deal with real data, but it could be used to get the output result schema. +* +* @param flatModelMapperClassName Name of the FlatModelMapper class +* @param modelScheme The schema of input Table type model. Review comment: it is not immediately clear to me what's the difference between `modelScheme` & `dataSchema`. Also why is modelScheme of the type `TableSchema` ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al…
walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al… URL: https://github.com/apache/flink/pull/9300#discussion_r324428983 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/mapper/FlatMapper.java ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.mapper; + +import org.apache.flink.ml.api.misc.param.Params; +import org.apache.flink.table.api.TableSchema; +import org.apache.flink.types.Row; +import org.apache.flink.util.Collector; + +import java.io.Serializable; + +/** + * Abstract class for flatMappers. + * FlatMapper transform one Row type data into zero, one, or more Row type result data. + * Operations that produce multiple strictly one Row type result data per Row type data + * can also use the {@link Mapper}. + */ +public abstract class FlatMapper implements Serializable { + +/** + * schema of the input. + */ + protected TableSchema dataSchema; + +/** + * params used for FlatMapper. + * User can set the params before the FlatMapper is executed. + */ + protected Params params; + + public FlatMapper(TableSchema dataSchema, Params params) { + this.dataSchema = dataSchema; + this.params = (null == params) ? new Params() : params.clone(); + } + + /** +* The core method of the FlatMapper. Takes an element from the input data set and transforms +* it into zero, one, or more elements. +* +* @param rowThe input row. +* @param output The collector for returning result values. +* @throws Exception This method may throw exceptions. Throwing an exception will cause the operation +* to fail. +*/ + public abstract void flatMap(Row row, Collector output) throws Exception; + +/** +* Wrapper method for the iterable input. +* @param rows the input rows. +* @param output the output collector +* @throws Exception if {@link #flatMap(Row, Collector)} throws exception. +*/ + public void flatMap(Iterable rows, Collector output) throws Exception { Review comment: in the JavaDoc it stated: "transform one Row of data into zero, one ore multiple ...", so is there a specific reason why this API is present? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al…
walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al… URL: https://github.com/apache/flink/pull/9300#discussion_r324428773 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/mapper/FlatMapper.java ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.mapper; + +import org.apache.flink.ml.api.misc.param.Params; +import org.apache.flink.table.api.TableSchema; +import org.apache.flink.types.Row; +import org.apache.flink.util.Collector; + +import java.io.Serializable; + +/** + * Abstract class for flatMappers. + * FlatMapper transform one Row type data into zero, one, or more Row type result data. + * Operations that produce multiple strictly one Row type result data per Row type data Review comment: this sentence is a bit confusing to me. not sure how to interpret "multiple stringly one row type result per row type data"? my understanding of your point is: `Mapper` is used for strictly zero or one output row, but flatMapper can support multiple row? please correct me if I am wrong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al…
walterddr commented on a change in pull request #9300: [FLINK-13513][ml] Add the FlatMapper and related classes for later al… URL: https://github.com/apache/flink/pull/9300#discussion_r324441096 ## File path: flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/mapper/FlatModelMapper.java ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.flink.ml.common.mapper; + +import org.apache.flink.ml.api.misc.param.Params; +import org.apache.flink.table.api.TableSchema; +import org.apache.flink.types.Row; + +import java.util.List; + +/** + * Abstract class for flatMappers with model. + * FlatModelMapper transform one Row type data into zero, one, or more Row type result data. Review comment: IMO, This is duplicate Javadoc similar to `FlatMapper`. Can we instead explain how the "model" part works. Basic `FlapMapper` already explains these. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-14076) 'ClassNotFoundException: KafkaException' on Flink v1.9 w/ checkpointing
Jeffrey Martin created FLINK-14076: -- Summary: 'ClassNotFoundException: KafkaException' on Flink v1.9 w/ checkpointing Key: FLINK-14076 URL: https://issues.apache.org/jira/browse/FLINK-14076 Project: Flink Issue Type: Bug Components: Connectors / Kafka Affects Versions: 1.9.0 Reporter: Jeffrey Martin Attachments: error.txt A Flink job that worked with checkpointing on a Flink v1.8.0 cluster fails on a Flink v1.9.0 cluster with checkpointing. It works on a Flink v1.9.0 cluster _without_ checkpointing. It is specifically _enabling checkpointing on v1.9.0_ that causes the JM to start throwing ClassNotFoundExceptions. Full stacktrace: [^error.txt] The job reads from Kafka via FlinkKafkaConsumer and writes to Kafka via FlinkKafkaProducer. The jobmanagers and taskmanagers are standalone. The exception is being raised deep in some Flink serialization code, so I'm not sure how to go about stepping through this in a debugger. The issue is happening in an internal repository at my job, but I can try to get a minimal repro on GitHub if it's not obvious from the error message alone what's broken. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate
flinkbot edited a comment on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate URL: https://github.com/apache/flink/pull/8468#issuecomment-524372146 ## CI report: * baae1632aabac35e6e08b402065857c4d67491f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124398771) * 6208617ff2d84bef7efaa7ee7cf96cba00031d88 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127279857) * 33997c30f049e32a22cd6caa0427568a52d25e63 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127650128) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14065) Log metric name when the metric fails on registration/unregistration
[ https://issues.apache.org/jira/browse/FLINK-14065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929784#comment-16929784 ] Zhilong Hong commented on FLINK-14065: -- Thanks, Till. I've already came up with a pull request. > Log metric name when the metric fails on registration/unregistration > > > Key: FLINK-14065 > URL: https://issues.apache.org/jira/browse/FLINK-14065 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.9.0 >Reporter: Zhilong Hong >Assignee: Zhilong Hong >Priority: Trivial > Labels: easyfix, pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When MetricGroup registers Metrics in MetricRegistryImpl, sometimes the > registration fails due to exceptions. However, currently it only logs > "{{Error while registering metric}}" with no more information, which is > inconvenient for users to troubleshoot which metric fails and why it fails. > Also, the warning log in registration and unregistration are both "{{Error > while registering metric}}". This will lead users to confusion (although > users can locate the correct place according to the call stack). > So I propose to log metric name when the metrics fails on > registration/unregistration. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate
flinkbot edited a comment on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate URL: https://github.com/apache/flink/pull/8468#issuecomment-524372146 ## CI report: * baae1632aabac35e6e08b402065857c4d67491f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124398771) * 6208617ff2d84bef7efaa7ee7cf96cba00031d88 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127279857) * 33997c30f049e32a22cd6caa0427568a52d25e63 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/127650128) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration
flinkbot edited a comment on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration URL: https://github.com/apache/flink/pull/9680#issuecomment-531487484 ## CI report: * 68fb3091be703893f8359ac99f04d29914557aa4 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127648210) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate
flinkbot edited a comment on issue #8468: [FLINK-12399][table][table-planner] Fix FilterableTableSource does not change after applyPredicate URL: https://github.com/apache/flink/pull/8468#issuecomment-524372146 ## CI report: * baae1632aabac35e6e08b402065857c4d67491f2 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124398771) * 6208617ff2d84bef7efaa7ee7cf96cba00031d88 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127279857) * 33997c30f049e32a22cd6caa0427568a52d25e63 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration
flinkbot edited a comment on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration URL: https://github.com/apache/flink/pull/9680#issuecomment-531487484 ## CI report: * 68fb3091be703893f8359ac99f04d29914557aa4 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/127648210) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration
flinkbot commented on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration URL: https://github.com/apache/flink/pull/9680#issuecomment-531487484 ## CI report: * 68fb3091be703893f8359ac99f04d29914557aa4 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration
flinkbot commented on issue #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration URL: https://github.com/apache/flink/pull/9680#issuecomment-531486535 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 68fb3091be703893f8359ac99f04d29914557aa4 (Sat Sep 14 15:01:03 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14065) Log metric name when the metric fails on registration/unregistration
[ https://issues.apache.org/jira/browse/FLINK-14065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14065: --- Labels: easyfix pull-request-available (was: easyfix) > Log metric name when the metric fails on registration/unregistration > > > Key: FLINK-14065 > URL: https://issues.apache.org/jira/browse/FLINK-14065 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.9.0 >Reporter: Zhilong Hong >Assignee: Zhilong Hong >Priority: Trivial > Labels: easyfix, pull-request-available > Fix For: 1.10.0 > > > When MetricGroup registers Metrics in MetricRegistryImpl, sometimes the > registration fails due to exceptions. However, currently it only logs > "{{Error while registering metric}}" with no more information, which is > inconvenient for users to troubleshoot which metric fails and why it fails. > Also, the warning log in registration and unregistration are both "{{Error > while registering metric}}". This will lead users to confusion (although > users can locate the correct place according to the call stack). > So I propose to log metric name when the metrics fails on > registration/unregistration. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] Thesharing opened a new pull request #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration
Thesharing opened a new pull request #9680: [FLINK-14065] Log metric name when the metric fails on registration/unregistration URL: https://github.com/apache/flink/pull/9680 ## What is the purpose of the change *This pull request adds metric name in log strings in registration and unregistration of metrics, so that users can troubleshoot which metric fails registering/unregistering itself.* ## Brief change log - *Add metric name in log string inside MetricRegistryImpl#register and MetricRegistryImpl#unregister* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14066) Pyflink building failure in master and 1.9.0 version
[ https://issues.apache.org/jira/browse/FLINK-14066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Yang updated FLINK-14066: Summary: Pyflink building failure in master and 1.9.0 version (was: pyflink building failure in master and 1.9.0 version) > Pyflink building failure in master and 1.9.0 version > > > Key: FLINK-14066 > URL: https://issues.apache.org/jira/browse/FLINK-14066 > Project: Flink > Issue Type: Bug > Components: API / Python, Build System >Affects Versions: 1.9.0, 1.10.0 > Environment: windows 10 enterprise x64(mentioned as build > environment, not development environment.) > powershell x64 > flink source master and 1.9.0 version > jdk-8u202 > maven-3.2.5 >Reporter: Xu Yang >Priority: Blocker > Labels: beginner, build > Attachments: setup.py > > Original Estimate: 1h > Remaining Estimate: 1h > > ATTENTION: This is a issue about building pyflink, not development. > During we build pyflink... > After we have built flink from flink source code, a folder named "target" is > generated. > Then, following the document description, "cd flink-python; python3 setup.py > sdist bdist_wheel", error happens. > Root cause: in the setup.py file, line 75, "FLINK_HOME = > os.path.abspath("../build-target")", the program can't found folder > "build-target", however, the building of flink generated a folder named > "target". So error happens in this way... > > The right way: > in ../flink-python/setup.py line 75, modify code as following: > FLINK_HOME = os.path.abspath("../target") -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9679: [hotfix] Fix the comments related to side output.
flinkbot edited a comment on issue #9679: [hotfix] Fix the comments related to side output. URL: https://github.com/apache/flink/pull/9679#issuecomment-531463762 ## CI report: * ca01daf37331ff493f367143af6a9a9c2cb28866 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/127638826) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9679: [hotfix] Fix the comments related to side output.
flinkbot edited a comment on issue #9679: [hotfix] Fix the comments related to side output. URL: https://github.com/apache/flink/pull/9679#issuecomment-531463762 ## CI report: * ca01daf37331ff493f367143af6a9a9c2cb28866 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/127638826) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9679: [hotfix] Fix the comments related to side output.
flinkbot commented on issue #9679: [hotfix] Fix the comments related to side output. URL: https://github.com/apache/flink/pull/9679#issuecomment-531463762 ## CI report: * ca01daf37331ff493f367143af6a9a9c2cb28866 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9679: [hotfix] Fix the comments related to side output.
flinkbot commented on issue #9679: [hotfix] Fix the comments related to side output. URL: https://github.com/apache/flink/pull/9679#issuecomment-531463292 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ca01daf37331ff493f367143af6a9a9c2cb28866 (Sat Sep 14 08:59:26 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] gaoyunhaii opened a new pull request #9679: [hotfix] Fix the comments related to side output.
gaoyunhaii opened a new pull request #9679: [hotfix] Fix the comments related to side output. URL: https://github.com/apache/flink/pull/9679 ## What is the purpose of the change The comments related to side output in the `Output#collect` method seem to be inconsistent with the logic. Therefore, this PR tries to address it. ## Brief change log - ca01daf37331ff493f367143af6a9a9c2cb28866 fixes the comments. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13748) Streaming File Sink s3 end-to-end test failed on Travis
[ https://issues.apache.org/jira/browse/FLINK-13748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929711#comment-16929711 ] Kostas Kloudas commented on FLINK-13748: Thanks [~till.rohrmann]. I will go with the first one then to see if it works. So far I was trying the second but I had tested it for Java 8 only. > Streaming File Sink s3 end-to-end test failed on Travis > --- > > Key: FLINK-13748 > URL: https://issues.apache.org/jira/browse/FLINK-13748 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem, Tests >Affects Versions: 1.10.0 >Reporter: Till Rohrmann >Assignee: Kostas Kloudas >Priority: Critical > Labels: test-stability > Fix For: 1.10.0 > > Attachments: image-2019-09-11-01-29-25-634.png > > > The {{Streaming File Sink s3 end-to-end test}} failed on Travis because it > did not produce any output for 10 minutes. > https://api.travis-ci.org/v3/job/572255913/log.txt -- This message was sent by Atlassian Jira (v8.3.2#803003)