[GitHub] [flink] flinkbot edited a comment on pull request #17810: [BP-1.13][FLINK-24631][k8s] Use a stable subset of labels to select jobManager and taskManager pods
flinkbot edited a comment on pull request #17810: URL: https://github.com/apache/flink/pull/17810#issuecomment-970348193 ## CI report: * 6feab49fae2754d12ad4d40a317cd9eeda8f46f2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26685) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16419) Avoid to recommit transactions which are known committed successfully to Kafka upon recovery
[
https://issues.apache.org/jira/browse/FLINK-16419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445711#comment-17445711
]
Fabian Paul commented on FLINK-16419:
-
[~qzhzm173227] every transaction is closed when the checkpoint succeeds and at
the beginning of a checkpoint a new transaction is started. As long as the
pipeline can make checkpoints the problem should theoretically not appear.
Can you double the transaction timeout configuration of your Kafka broker [1]?
If either the transaction timeout of the producer or the transaction timeout of
your broker is lower than the checkpoint interval you'll see transaction
timeouts.
[1]
https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#brokerconfigs_transaction.max.timeout.ms
> Avoid to recommit transactions which are known committed successfully to
> Kafka upon recovery
>
>
> Key: FLINK-16419
> URL: https://issues.apache.org/jira/browse/FLINK-16419
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Runtime / Checkpointing
>Reporter: Jun Qin
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor, usability
>
> When recovering from a snapshot (checkpoint/savepoint), FlinkKafkaProducer
> tries to recommit all pre-committed transactions which are in the snapshot,
> even if those transactions were successfully committed before (i.e., the call
> to {{kafkaProducer.commitTransaction()}} via {{notifyCheckpointComplete()}}
> returns OK). This may lead to recovery failures when recovering from a very
> old snapshot because the transactional IDs in that snapshot may have been
> expired and removed from Kafka. For example the following scenario:
> # Start a Flink job with FlinkKafkaProducer sink with exactly-once
> # Suspend the Flink job with a savepoint A
> # Wait for time longer than {{transactional.id.expiration.ms}} +
> {{transaction.remove.expired.transaction.cleanup.interval.ms}}
> # Recover the job with savepoint A.
> # The recovery will fail with the following error:
> {noformat}
> 2020-02-26 14:33:25,817 INFO
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer
> - Attempting to resume transaction Source: Custom Source -> Sink:
> Unnamed-7df19f87deec5680128845fd9a6ca18d-1 with producerId 2001 and epoch
> 1202020-02-26 14:33:25,914 INFO org.apache.kafka.clients.Metadata
> - Cluster ID: RN0aqiOwTUmF5CnHv_IPxA
> 2020-02-26 14:33:26,017 INFO org.apache.kafka.clients.producer.KafkaProducer
> - [Producer clientId=producer-1, transactionalId=Source: Custom
> Source -> Sink: Unnamed-7df19f87deec5680128845fd9a6ca18d-1] Closing the Kafka
> producer with timeoutMillis = 92233720
> 36854775807 ms.
> 2020-02-26 14:33:26,019 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source -> Sink: Unnamed (1/1)
> (a77e457941f09cd0ebbd7b982edc0f02) switched from RUNNING to FAILED.
> org.apache.kafka.common.KafkaException: Unhandled error in EndTxnResponse:
> The producer attempted to use a producer id which is not currently assigned
> to its transactional id.
> at
> org.apache.kafka.clients.producer.internals.TransactionManager$EndTxnHandler.handleResponse(TransactionManager.java:1191)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:909)
> at
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> at
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:557)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:288)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For now, the workaround is to call
> {{producer.ignoreFailuresAfterTransactionTimeout()}}. This is a bit risky, as
> it may hide real transaction timeout errors.
> After discussed with [~becket_qin], [~pnowojski] and [~aljoscha], a possible
> way is to let JobManager, after successfully notifies all operators the
> completion of a snapshot (via {{notifyCheckpoingComplete}}), record the
> success, e.g., write the successful transactional IDs somewhere in the
> snapshot. Then those transactions need not recommit upon recovery.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-24950) Use Hive Dialect execute Hive DDL, But throw a NullPointerException
xiaojunchen created FLINK-24950:
---
Summary: Use Hive Dialect execute Hive DDL, But throw a
NullPointerException
Key: FLINK-24950
URL: https://issues.apache.org/jira/browse/FLINK-24950
Project: Flink
Issue Type: Bug
Components: Connectors / Hive
Affects Versions: 1.13.2
Environment: flink-1.13.2
cdh5.14.2
jdk8
Reporter: xiaojunchen
Dear all friends:
I try to execute a hive ddl sql with stream table api on flink-1.13.2, the code
like:
```java
String hiveDDL = ResourceUtil.readClassPathSource("hive-ddl.sql");
EnvironmentSettings settings = EnvironmentSettings.newInstance()
.useBlinkPlanner()
.inStreamingMode().build();
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings);
String name = "hive";
String defaultDatabase = "stream";
String hiveConfDir = "conf";
HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir);
tableEnv.registerCatalog("hive", hive);
tableEnv.useCatalog("hive");
tableEnv.useDatabase("stream");
tableEnv.executeSql("DROP TABLE IF EXISTS dimension_table");
// 设置HIVE方言
tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
tableEnv.executeSql(hiveDDL);
```
the hive server in cdh5.14.2, and the ddl sql like:
```sql
CREATE TABLE dimension_table (
product_id STRING,
product_name STRING,
unit_price DECIMAL(10, 4),
pv_count BIGINT,
like_count BIGINT,
comment_count BIGINT,
update_time TIMESTAMP(3),
update_user STRING
)
PARTITIONED BY (
pt_year STRING,
pt_month STRING,
pt_day STRING
)
TBLPROPERTIES (
-- using default partition-name order to load the latest partition every 12h
(the most recommended and convenient way)
'streaming-source.enable' = 'true',
'streaming-source.partition.include' = 'latest',
'streaming-source.monitor-interval' = '12 h',
'streaming-source.partition-order' = 'partition-name', -- option with default
value, can be ignored.
-- using partition file create-time order to load the latest partition every 12h
'streaming-source.enable' = 'true',
'streaming-source.partition.include' = 'latest',
'streaming-source.partition-order' = 'create-time',
'streaming-source.monitor-interval' = '12 h'
-- using partition-time order to load the latest partition every 12h
'streaming-source.enable' = 'true',
'streaming-source.partition.include' = 'latest',
'streaming-source.monitor-interval' = '12 h',
'streaming-source.partition-order' = 'partition-time',
'partition.time-extractor.kind' = 'default',
'partition.time-extractor.timestamp-pattern' = '$pt_year-$pt_month-$pt_day
00:00:00'
)
```
then run it, but throw NullPointerException, like:
```
2021-11-18 15:33:00,387 INFO [org.apache.flink.table.catalog.hive.HiveCatalog]
- Setting hive conf dir as conf
2021-11-18 15:33:00,481 WARN [org.apache.hadoop.util.NativeCodeLoader] - Unable
to load native-hadoop library for your platform... using builtin-java classes
where applicable
2021-11-18 15:33:01,345 INFO [org.apache.flink.table.catalog.hive.HiveCatalog]
- Created HiveCatalog 'hive'
2021-11-18 15:33:01,371 INFO [hive.metastore] - Trying to connect to metastore
with URI thrift://cdh-dev-node-119:9083
2021-11-18 15:33:01,441 INFO [hive.metastore] - Opened a connection to
metastore, current connections: 1
2021-11-18 15:33:01,521 INFO [hive.metastore] - Connected to metastore.
2021-11-18 15:33:01,856 INFO [org.apache.flink.table.catalog.hive.HiveCatalog]
- Connected to Hive metastore
2021-11-18 15:33:01,899 INFO [org.apache.flink.table.catalog.CatalogManager] -
Set the current default catalog as [hive] and the current default database as
[stream].
2021-11-18 15:33:03,290 INFO [org.apache.hadoop.hive.ql.session.SessionState] -
Created local directory:
/var/folders/4m/n1wgh7rd2yqfv301kq00l4q4gn/T/681dd0aa-ba35-4a0e-b069-3ad48f030774_resources
2021-11-18 15:33:03,298 INFO [org.apache.hadoop.hive.ql.session.SessionState] -
Created HDFS directory:
/tmp/hive/chenxiaojun/681dd0aa-ba35-4a0e-b069-3ad48f030774
2021-11-18 15:33:03,305 INFO [org.apache.hadoop.hive.ql.session.SessionState] -
Created local directory:
/var/folders/4m/n1wgh7rd2yqfv301kq00l4q4gn/T/chenxiaojun/681dd0aa-ba35-4a0e-b069-3ad48f030774
2021-11-18 15:33:03,311 INFO [org.apache.hadoop.hive.ql.session.SessionState] -
Created HDFS directory:
/tmp/hive/chenxiaojun/681dd0aa-ba35-4a0e-b069-3ad48f030774/_tmp_space.db
2021-11-18 15:33:03,314 INFO [org.apache.hadoop.hive.ql.session.SessionState] -
No Tez session required at this point. hive.execution.engine=mr.
Exception in thread "main" java.lang.NullPointerException
at
org.apache.flink.table.catalog.hive.client.HiveShimV100.registerTemporaryFunction(HiveShimV100.java:422)
at
org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:217)
at
[GitHub] [flink] AHeise commented on a change in pull request #17792: [FLINK-24763][fs-connector] LimitableReader should swallow exception when reached limit
AHeise commented on a change in pull request #17792:
URL: https://github.com/apache/flink/pull/17792#discussion_r751969408
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/LimitableBulkFormat.java
##
@@ -105,8 +111,16 @@ private boolean reachLimit() {
return null;
}
-RecordIterator batch = reader.readBatch();
-return batch == null ? null : new LimitableIterator(batch);
+try {
+RecordIterator batch = reader.readBatch();
+return batch == null ? null : new LimitableIterator(batch);
+} catch (Exception e) {
Review comment:
This change is a no-op in a sequential threading model. If
`reachLimit()` returns true, then the first `if` in `readBatch` already returns
`null`. In all other cases, the exception is rethrown.
So I'm assuming you are actually guarding against some concurrent
modification in another thread. If so, then I'd rather fix the threading model.
This class already uses two different mechanisms of dealing with concurrency
(`synchronized` and `AtomicLong`) and you now add optimistic invocation with
exception handling as a third. This is a hard to reason. You probably should
settle with using a `Lock` and use it all the way.
Btw I don't understand many parts of this class:
- What's the purpose of `globalNumberRead`? The bulk format instance will be
replicated per subtask, so why would you need it?
- Do you ensure that only one subtask is executed or how do you consolidate
the different lists of the different subtask? `LIMIT N` would produce a list of
`N` records in each subtask, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-10737) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445703#comment-17445703
]
Yun Gao commented on FLINK-10737:
-
{code:java}
Nov 18 00:02:56 [ERROR] Tests run: 15, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 275.977 s <<< FAILURE! - in
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
Nov 18 00:02:56 [ERROR] testScaleDownBeforeFirstCheckpoint Time elapsed:
64.757 s <<< ERROR!
Nov 18 00:02:56 org.apache.kafka.common.errors.TimeoutException:
org.apache.kafka.common.errors.TimeoutException: Timeout expired after
6milliseconds while awaiting InitProducerId
Nov 18 00:02:56 Caused by: org.apache.kafka.common.errors.TimeoutException:
Timeout expired after 6milliseconds while awaiting InitProducerId
Nov 18 00:02:56
Nov 18 00:02:56 [INFO]
Nov 18 00:02:56 [INFO] Results:
Nov 18 00:02:56 [INFO]
Nov 18 00:02:56 [ERROR] Errors:
Nov 18 00:02:56 [ERROR]
FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint » Timeout
org.apac...
Nov 18 00:02:56 [INFO]
Nov 18 00:02:56 [ERROR] Tests run: 193, Failures: 0, Errors: 1, Skipped: 0
{code}
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26679=logs=1fc6e7bf-633c-5081-c32a-9dea24b05730=576aba0a-d787-51b6-6a92-cf233f360582=7207]
Hi [~becket_qin] This seems to be another issue that reproduced after a very
long time...
> FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis
>
>
> Key: FLINK-10737
> URL: https://issues.apache.org/jira/browse/FLINK-10737
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.7.0, 1.8.0, 1.14.1
>Reporter: Till Rohrmann
>Assignee: Jiangjie Qin
>Priority: Critical
> Labels: test-stability
>
> The {{FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint}} failed on
> Travis:
> https://api.travis-ci.org/v3/job/448781612/log.txt
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] Airblader commented on pull request #17808: [FLINK-24928][flink-runtime-web] Typing improvements for Flink UI
Airblader commented on pull request #17808: URL: https://github.com/apache/flink/pull/17808#issuecomment-972611394 @AHeise And the CI has passed again now, too. :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal edited a comment on pull request #17692: [FLINK-24489][CEP] The size of entryCache & eventsBufferCache in the SharedBuffer should be defined with a threshold to limit the n
RocMarshal edited a comment on pull request #17692: URL: https://github.com/apache/flink/pull/17692#issuecomment-969035044 > The situation is a bit odd. I think we shouldn't have a cache on top of rocks DB at all. We could think about having a cache in rocks DB itself. > > Nevertheless, the cache is there and causes trouble for you. I understood now that your changes are not impacting correctness but only performance, so I think your contribution certainly is welcome. > > To reiterate from my first rough review: Could you please add a (unit) test and make the configuration descriptions easier to understand for folks that have not read the CEP code? Thanks so much for your @AHeise reply and explanation, which makes me suddenly see the light. As updated in the `CEPCacheOptions`: ` And it could accelerate the CEP operate process speed and limit the capacity of cache in pure memory. Note: It's only effective to limit usage of memory when 'state.backend' was set as 'rocksdb', which would transport the elements exceeded the number of the cache into the rocksdb state storage instead of memory state storage. ` The minor pr is really helpful for memory limitation when the `state.backend` is set as `rocksdb`. By contrast，when the `state.backend` is set as not `rocksdb`, the current cache in the pr would cause performance decreased. Compared with old cache , the state part will contain more elements swapped out from new guava-cache ,which would make it heavier to `copy on write` for state . In my limited read, if the positive significance of the pr is bigger than the side-effects of the pr, it would be worthy to do. And Please let me know what do you think of it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24949) KafkaITCase.testBigRecordJob fails on azure
[
https://issues.apache.org/jira/browse/FLINK-24949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445695#comment-17445695
]
Yun Gao commented on FLINK-24949:
-
Hi [~becket_qin] This issue seems to reproduced after fixed for a long time, is
this the same problem with prior one~?
> KafkaITCase.testBigRecordJob fails on azure
> ---
>
> Key: FLINK-24949
> URL: https://issues.apache.org/jira/browse/FLINK-24949
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.1
>
>
> {code:java}
> Nov 17 23:39:39 [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 222.57 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.KafkaITCase
> Nov 17 23:39:39 [ERROR] testBigRecordJob Time elapsed: 60.02 s <<< ERROR!
> Nov 17 23:39:39 org.junit.runners.model.TestTimedOutException: test timed out
> after 6 milliseconds
> Nov 17 23:39:39 at sun.misc.Unsafe.park(Native Method)
> Nov 17 23:39:39 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> Nov 17 23:39:39 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Nov 17 23:39:39 at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:58)
> Nov 17 23:39:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBigRecordTestTopology(KafkaConsumerTestBase.java:1473)
> Nov 17 23:39:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testBigRecordJob(KafkaITCase.java:119)
> Nov 17 23:39:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 17 23:39:39 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 17 23:39:39 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 17 23:39:39 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 17 23:39:39 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 17 23:39:39 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 17 23:39:39 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Nov 17 23:39:39 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Nov 17 23:39:39 at java.lang.Thread.run(Thread.java:748)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26679=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=7161
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24949) KafkaITCase.testBigRecordJob fails on azure
[
https://issues.apache.org/jira/browse/FLINK-24949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24949:
Affects Version/s: 1.14.1
> KafkaITCase.testBigRecordJob fails on azure
> ---
>
> Key: FLINK-24949
> URL: https://issues.apache.org/jira/browse/FLINK-24949
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.1
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.1
>
>
> {code:java}
> Nov 17 23:39:39 [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 222.57 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.KafkaITCase
> Nov 17 23:39:39 [ERROR] testBigRecordJob Time elapsed: 60.02 s <<< ERROR!
> Nov 17 23:39:39 org.junit.runners.model.TestTimedOutException: test timed out
> after 6 milliseconds
> Nov 17 23:39:39 at sun.misc.Unsafe.park(Native Method)
> Nov 17 23:39:39 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> Nov 17 23:39:39 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Nov 17 23:39:39 at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:58)
> Nov 17 23:39:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBigRecordTestTopology(KafkaConsumerTestBase.java:1473)
> Nov 17 23:39:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testBigRecordJob(KafkaITCase.java:119)
> Nov 17 23:39:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 17 23:39:39 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 17 23:39:39 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 17 23:39:39 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 17 23:39:39 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 17 23:39:39 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 17 23:39:39 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Nov 17 23:39:39 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Nov 17 23:39:39 at java.lang.Thread.run(Thread.java:748)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26679=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=7161
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Reopened] (FLINK-10737) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao reopened FLINK-10737:
-
> FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis
>
>
> Key: FLINK-10737
> URL: https://issues.apache.org/jira/browse/FLINK-10737
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.7.0, 1.8.0, 1.14.1
>Reporter: Till Rohrmann
>Assignee: Jiangjie Qin
>Priority: Critical
> Labels: test-stability
>
> The {{FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint}} failed on
> Travis:
> https://api.travis-ci.org/v3/job/448781612/log.txt
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-10737) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-10737:
Affects Version/s: 1.14.1
> FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis
>
>
> Key: FLINK-10737
> URL: https://issues.apache.org/jira/browse/FLINK-10737
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.7.0, 1.8.0, 1.14.1
>Reporter: Till Rohrmann
>Assignee: Jiangjie Qin
>Priority: Critical
> Labels: test-stability
>
> The {{FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint}} failed on
> Travis:
> https://api.travis-ci.org/v3/job/448781612/log.txt
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24936) Python test_data_stream.py hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-24936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445689#comment-17445689
]
Yun Gao commented on FLINK-24936:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26679=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23892]
> Python test_data_stream.py hangs on azure
> -
>
> Key: FLINK-24936
> URL: https://issues.apache.org/jira/browse/FLINK-24936
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Assignee: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 17 04:36:57 pyflink/datastream/tests/test_data_stream.py
> ... [ 33%]
> Nov 17 04:52:58
> ==
> Nov 17 04:52:58 Process produced no output for 900 seconds.
> Nov 17 04:52:58
> ==
> Nov 17 04:52:58
> ==
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26627=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17685: [FLINK-24631][Kubernetes]Use minimal selector to select jobManager and taskManager pod
flinkbot edited a comment on pull request #17685: URL: https://github.com/apache/flink/pull/17685#issuecomment-961148655 ## CI report: * 8c1def7a1d20b4440e0d6fdb3b9beeeb200e85f8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26683) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24949) KafkaITCase.testBigRecordJob fails on azure
Yun Gao created FLINK-24949:
---
Summary: KafkaITCase.testBigRecordJob fails on azure
Key: FLINK-24949
URL: https://issues.apache.org/jira/browse/FLINK-24949
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Reporter: Yun Gao
Fix For: 1.14.1
{code:java}
Nov 17 23:39:39 [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 222.57 s <<< FAILURE! - in
org.apache.flink.streaming.connectors.kafka.KafkaITCase
Nov 17 23:39:39 [ERROR] testBigRecordJob Time elapsed: 60.02 s <<< ERROR!
Nov 17 23:39:39 org.junit.runners.model.TestTimedOutException: test timed out
after 6 milliseconds
Nov 17 23:39:39 at sun.misc.Unsafe.park(Native Method)
Nov 17 23:39:39 at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
Nov 17 23:39:39 at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
Nov 17 23:39:39 at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
Nov 17 23:39:39 at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
Nov 17 23:39:39 at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Nov 17 23:39:39 at
org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:58)
Nov 17 23:39:39 at
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBigRecordTestTopology(KafkaConsumerTestBase.java:1473)
Nov 17 23:39:39 at
org.apache.flink.streaming.connectors.kafka.KafkaITCase.testBigRecordJob(KafkaITCase.java:119)
Nov 17 23:39:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Nov 17 23:39:39 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Nov 17 23:39:39 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Nov 17 23:39:39 at java.lang.reflect.Method.invoke(Method.java:498)
Nov 17 23:39:39 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Nov 17 23:39:39 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Nov 17 23:39:39 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
Nov 17 23:39:39 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Nov 17 23:39:39 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
Nov 17 23:39:39 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
Nov 17 23:39:39 at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Nov 17 23:39:39 at java.lang.Thread.run(Thread.java:748)
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26679=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=7161
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17809: [BP-1.14][FLINK-24631][k8s] Use a stable subset of labels to select jobManager and taskManager pods
flinkbot edited a comment on pull request #17809: URL: https://github.com/apache/flink/pull/17809#issuecomment-970332023 ## CI report: * 6c854a3c03d213703bb94d904e822b2239deef1c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26684) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24948) Special character in column names breaks JDBC statement parsing
[ https://issues.apache.org/jira/browse/FLINK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445685#comment-17445685 ] Paul Lin commented on FLINK-24948: -- Working on the fix. Please assign this issue to me. > Special character in column names breaks JDBC statement parsing > --- > > Key: FLINK-24948 > URL: https://issues.apache.org/jira/browse/FLINK-24948 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.4 >Reporter: Paul Lin >Priority: Major > > Currently, JDBC connector assumes columns names respect Java identifier > naming restrictions, but Databases that support JDBC may have different > naming restrictions. For example, MySQL allows dots and colons in column > names. In that case, JDBC connector would have trouble parsing the SQL. > We could fix this by validating field names in `JdbcDmlOptions`. In addition, > it'd be good to clarify the naming restrictions of Flink SQL, so users and > connector developers would know the standard. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24948) Special character in column names breaks JDBC statement parsing
Paul Lin created FLINK-24948: Summary: Special character in column names breaks JDBC statement parsing Key: FLINK-24948 URL: https://issues.apache.org/jira/browse/FLINK-24948 Project: Flink Issue Type: Bug Components: Connectors / JDBC Affects Versions: 1.12.4 Reporter: Paul Lin Currently, JDBC connector assumes columns names respect Java identifier naming restrictions, but Databases that support JDBC may have different naming restrictions. For example, MySQL allows dots and colons in column names. In that case, JDBC connector would have trouble parsing the SQL. We could fix this by validating field names in `JdbcDmlOptions`. In addition, it'd be good to clarify the naming restrictions of Flink SQL, so users and connector developers would know the standard. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24947) Flink on k8s support HostNetWork model
[ https://issues.apache.org/jira/browse/FLINK-24947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445684#comment-17445684 ] liuzhuo commented on FLINK-24947: - Within our company, we have completed the implementation of HostNetWork and it is already up and running. If necessary, I would be happy to contribute this part of the function > Flink on k8s support HostNetWork model > -- > > Key: FLINK-24947 > URL: https://issues.apache.org/jira/browse/FLINK-24947 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: liuzhuo >Priority: Minor > > For the use of flink on k8s, for performance considerations, it is important > to choose a CNI plug-in. Usually we have two environments: Managed and > UnManaged. > Managed: Cloud vendors usually provide very efficient CNI plug-ins, we > don’t need to care about network performance issues > UnManaged: On self-built K8s clusters, CNI plug-ins are usually optional, > similar to Flannel and Calcico, but such software network cards usually lose > some performance or require some additional network strategies. > For an unmanaged environment, if we also want to achieve the best network > performance, should we support the *HostNetWork* model? > Use the host network to achieve the best performance -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24947) Flink on k8s support HostNetWork model
[ https://issues.apache.org/jira/browse/FLINK-24947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuzhuo updated FLINK-24947: Attachment: image-2021-11-18-15-13-20-370.png > Flink on k8s support HostNetWork model > -- > > Key: FLINK-24947 > URL: https://issues.apache.org/jira/browse/FLINK-24947 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: liuzhuo >Priority: Minor > > For the use of flink on k8s, for performance considerations, it is important > to choose a CNI plug-in. Usually we have two environments: Managed and > UnManaged. > Managed: Cloud vendors usually provide very efficient CNI plug-ins, we > don’t need to care about network performance issues > UnManaged: On self-built K8s clusters, CNI plug-ins are usually optional, > similar to Flannel and Calcico, but such software network cards usually lose > some performance or require some additional network strategies. > For an unmanaged environment, if we also want to achieve the best network > performance, should we support the *HostNetWork* model? > Use the host network to achieve the best performance -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24947) Flink on k8s support HostNetWork model
[ https://issues.apache.org/jira/browse/FLINK-24947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuzhuo updated FLINK-24947: Attachment: (was: image-2021-11-18-15-13-20-370.png) > Flink on k8s support HostNetWork model > -- > > Key: FLINK-24947 > URL: https://issues.apache.org/jira/browse/FLINK-24947 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: liuzhuo >Priority: Minor > > For the use of flink on k8s, for performance considerations, it is important > to choose a CNI plug-in. Usually we have two environments: Managed and > UnManaged. > Managed: Cloud vendors usually provide very efficient CNI plug-ins, we > don’t need to care about network performance issues > UnManaged: On self-built K8s clusters, CNI plug-ins are usually optional, > similar to Flannel and Calcico, but such software network cards usually lose > some performance or require some additional network strategies. > For an unmanaged environment, if we also want to achieve the best network > performance, should we support the *HostNetWork* model? > Use the host network to achieve the best performance -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24947) Flink on k8s support HostNetWork model
liuzhuo created FLINK-24947: --- Summary: Flink on k8s support HostNetWork model Key: FLINK-24947 URL: https://issues.apache.org/jira/browse/FLINK-24947 Project: Flink Issue Type: New Feature Components: Deployment / Kubernetes Reporter: liuzhuo For the use of flink on k8s, for performance considerations, it is important to choose a CNI plug-in. Usually we have two environments: Managed and UnManaged. Managed: Cloud vendors usually provide very efficient CNI plug-ins, we don’t need to care about network performance issues UnManaged: On self-built K8s clusters, CNI plug-ins are usually optional, similar to Flannel and Calcico, but such software network cards usually lose some performance or require some additional network strategies. For an unmanaged environment, if we also want to achieve the best network performance, should we support the *HostNetWork* model? Use the host network to achieve the best performance -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24935) Python module failed to compile due to "Could not create local repository"
[
https://issues.apache.org/jira/browse/FLINK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445683#comment-17445683
]
Yun Gao commented on FLINK-24935:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26677=logs=a29bcfe1-064d-50b9-354f-07802213a3c0=47ff6576-c9dc-5eab-9db8-183dcca3bede=28]
> Python module failed to compile due to "Could not create local repository"
> --
>
> Key: FLINK-24935
> URL: https://issues.apache.org/jira/browse/FLINK-24935
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.12.5
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Invoking mvn with 'mvn -Dmaven.wagon.http.pool=false --settings
> /__w/1/s/tools/ci/google-mirror-settings.xml
> -Dorg.slf4j.simpleLogger.showDateTime=true
> -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS
> -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
> --no-snapshot-updates -B -Dhadoop.version=2.8.3 -Dinclude_hadoop_aws
> -Dscala-2.11 clean deploy
> -DaltDeploymentRepository=validation_repository::default::file:/tmp/flink-validation-deployment
> -Dmaven.repo.local=/home/vsts/work/1/.m2/repository
> -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2
> -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -U -DskipTests'
> [ERROR] Could not create local repository at /home/vsts/work/1/.m2/repository
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26625=logs=a29bcfe1-064d-50b9-354f-07802213a3c0=47ff6576-c9dc-5eab-9db8-183dcca3bede]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24763) ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445682#comment-17445682
]
Yun Gao commented on FLINK-24763:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26676=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=2e426bf0-b717-56bb-ab62-d63086457354=13523]
> ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
> ---
>
> Key: FLINK-24763
> URL: https://issues.apache.org/jira/browse/FLINK-24763
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Jingsong Lee
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> The test {{ParquetFileSystemITCase.testLimitableBulkFormat}} fails with
> {code}
> 2021-11-03T22:10:11.5106075Z Nov 03 22:10:11 [ERROR]
> testLimitableBulkFormat[false] Time elapsed: 9.177 s <<< ERROR!
> 2021-11-03T22:10:11.5106643Z Nov 03 22:10:11 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-11-03T22:10:11.5107213Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-11-03T22:10:11.5111034Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-11-03T22:10:11.5112190Z Nov 03 22:10:11 at
> org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:188)
> 2021-11-03T22:10:11.5112892Z Nov 03 22:10:11 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2021-11-03T22:10:11.5113393Z Nov 03 22:10:11 at
> org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
> 2021-11-03T22:10:11.5114157Z Nov 03 22:10:11 at
> org.apache.flink.formats.parquet.ParquetFileSystemITCase.testLimitableBulkFormat(ParquetFileSystemITCase.java:128)
> 2021-11-03T22:10:11.5114951Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-11-03T22:10:11.5115568Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-11-03T22:10:11.5116115Z Nov 03 22:10:11 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-11-03T22:10:11.5116591Z Nov 03 22:10:11 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-11-03T22:10:11.5117088Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-11-03T22:10:11.5117807Z Nov 03 22:10:11 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-11-03T22:10:11.5118821Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-11-03T22:10:11.5119417Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-11-03T22:10:11.5119944Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-11-03T22:10:11.5120427Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-11-03T22:10:11.5120919Z Nov 03 22:10:11 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-11-03T22:10:11.5121571Z Nov 03 22:10:11 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-11-03T22:10:11.5122526Z Nov 03 22:10:11 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-03T22:10:11.5123245Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-11-03T22:10:11.5123804Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-11-03T22:10:11.5124314Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-11-03T22:10:11.5124806Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-11-03T22:10:11.5125313Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-11-03T22:10:11.5125810Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-11-03T22:10:11.5126281Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-11-03T22:10:11.5126739Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-11-03T22:10:11.5127349Z Nov 03 22:10:11 at
>
[jira] [Commented] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445681#comment-17445681
]
Yun Gao commented on FLINK-24495:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26676=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23594]
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24945) The exception location on the WebUI should contain TM's ResourceID
[ https://issues.apache.org/jira/browse/FLINK-24945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Wang updated FLINK-24945: --- Description: Currently, the exception location on WebUI only contains hostname and port, as shown below: !image-2021-11-18-14-40-39-855.png|width=907,height=267! But in fact, we usually use ResourceID (ContainerID on yarn, PodID on kubernetes) to locate the corresponding TM instead of hostname+port, so I think the ResourceID should also be displayed here, which is more convenient for users. was: Currently, the exception location on WebUI only contains hostname and port, as shown below: !image-2021-11-18-14-40-39-855.png|width=638,height=188! But in fact, we usually use ResourceID (ContainerID on yarn, PodID on kubernetes) to locate the corresponding TM instead of hostname+port, so I think the ResourceID should also be displayed here, which is more convenient for users. > The exception location on the WebUI should contain TM's ResourceID > --- > > Key: FLINK-24945 > URL: https://issues.apache.org/jira/browse/FLINK-24945 > Project: Flink > Issue Type: Improvement >Reporter: Lijie Wang >Priority: Major > Attachments: image-2021-11-18-14-40-39-855.png > > > Currently, the exception location on WebUI only contains hostname and port, > as shown below: > !image-2021-11-18-14-40-39-855.png|width=907,height=267! > But in fact, we usually use ResourceID (ContainerID on yarn, PodID on > kubernetes) to locate the corresponding TM instead of hostname+port, so I > think the ResourceID should also be displayed here, which is more convenient > for users. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24946) KafkaSourceLegacyITCase.testBigRecordJob fails on azure
Yun Gao created FLINK-24946:
---
Summary: KafkaSourceLegacyITCase.testBigRecordJob fails on azure
Key: FLINK-24946
URL: https://issues.apache.org/jira/browse/FLINK-24946
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
Nov 17 16:43:38 at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
Nov 17 16:43:38 at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
Nov 17 16:43:38 at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
Nov 17 16:43:38 at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
Nov 17 16:43:38 at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
Nov 17 16:43:38 at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
Nov 17 16:43:38 at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
Nov 17 16:43:38 at
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
Nov 17 16:43:38 at
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
Nov 17 16:43:38 at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
Nov 17 16:43:38 at
org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
Nov 17 16:43:38 at
org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
Nov 17 16:43:38 at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
Nov 17 16:43:38 at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
Nov 17 16:43:38 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
Nov 17 16:43:38 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
Nov 17 16:43:38 at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
Nov 17 16:43:38 at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
Nov 17 16:43:38 at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
Nov 17 16:43:38 at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Nov 17 16:43:38
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26672=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=6735
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24800) BufferTimeoutITCase.testDisablingBufferTimeout failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445676#comment-17445676
]
Yun Gao commented on FLINK-24800:
-
Very thanks [~akalashnikov] for backporting the fix! I'll try having a look~
> BufferTimeoutITCase.testDisablingBufferTimeout failed on Azure
> --
>
> Key: FLINK-24800
> URL: https://issues.apache.org/jira/browse/FLINK-24800
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.13.3, 1.15.0
>Reporter: Yun Gao
>Assignee: Anton Kalashnikov
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> {code:java}
> 2021-11-05T12:18:50.5272055Z Nov 05 12:18:50 [INFO] Results:
> 2021-11-05T12:18:50.5273369Z Nov 05 12:18:50 [INFO]
> 2021-11-05T12:18:50.5274011Z Nov 05 12:18:50 [ERROR] Failures:
> 2021-11-05T12:18:50.5274518Z Nov 05 12:18:50 [ERROR]
> BufferTimeoutITCase.testDisablingBufferTimeout:85
> 2021-11-05T12:18:50.5274871Z Nov 05 12:18:50 Expected: <0>
> 2021-11-05T12:18:50.5275150Z Nov 05 12:18:50 but: was <1>
> 2021-11-05T12:18:50.5276136Z Nov 05 12:18:50 [INFO]
> 2021-11-05T12:18:50.5276667Z Nov 05 12:18:50 [ERROR] Tests run: 1849,
> Failures: 1, Errors: 0, Skipped: 58
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26018=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=10850
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24945) The exception location on the WebUI should contain TM's ResourceID
[ https://issues.apache.org/jira/browse/FLINK-24945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Wang updated FLINK-24945: --- Description: Currently, the exception location on WebUI only contains hostname and port, as shown below: !image-2021-11-18-14-40-39-855.png|width=638,height=188! But in fact, we usually use ResourceID (ContainerID on yarn, PodID on kubernetes) to locate the corresponding TM instead of hostname+port, so I think the ResourceID should also be displayed here, which is more convenient for users. was: Currently, the exception location on WebUI only contains hostname and port, as shown below: !image-2021-11-18-14-40-39-855.png|width=638,height=188! But in fact, we usually use ResourceID (ContainerID on yarn, PodID on kubernetes) to locate the corresponding TM instead of hostname+port, so I think the resourceID should also be displayed here, which is more convenient for users. > The exception location on the WebUI should contain TM's ResourceID > --- > > Key: FLINK-24945 > URL: https://issues.apache.org/jira/browse/FLINK-24945 > Project: Flink > Issue Type: Improvement >Reporter: Lijie Wang >Priority: Major > Attachments: image-2021-11-18-14-40-39-855.png > > > Currently, the exception location on WebUI only contains hostname and port, > as shown below: > !image-2021-11-18-14-40-39-855.png|width=638,height=188! > But in fact, we usually use ResourceID (ContainerID on yarn, PodID on > kubernetes) to locate the corresponding TM instead of hostname+port, so I > think the ResourceID should also be displayed here, which is more convenient > for users. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24945) The exception location on the WebUI should contain TM's ResourceID
Lijie Wang created FLINK-24945: -- Summary: The exception location on the WebUI should contain TM's ResourceID Key: FLINK-24945 URL: https://issues.apache.org/jira/browse/FLINK-24945 Project: Flink Issue Type: Improvement Reporter: Lijie Wang Attachments: image-2021-11-18-14-40-39-855.png Currently, the exception location on WebUI only contains hostname and port, as shown below: !image-2021-11-18-14-40-39-855.png|width=638,height=188! But in fact, we usually use ResourceID (ContainerID on yarn, PodID on kubernetes) to locate the corresponding TM instead of hostname+port, so I think the resourceID should also be displayed here, which is more convenient for users. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24936) Python test_data_stream.py hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-24936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445672#comment-17445672
]
Yun Gao commented on FLINK-24936:
-
Got that, very thanks [~hxbks2ks] for attending the issue!
> Python test_data_stream.py hangs on azure
> -
>
> Key: FLINK-24936
> URL: https://issues.apache.org/jira/browse/FLINK-24936
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Assignee: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 17 04:36:57 pyflink/datastream/tests/test_data_stream.py
> ... [ 33%]
> Nov 17 04:52:58
> ==
> Nov 17 04:52:58 Process produced no output for 900 seconds.
> Nov 17 04:52:58
> ==
> Nov 17 04:52:58
> ==
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26627=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24944) Flink job on k8s mode supports pod scheduling according to priority
[
https://issues.apache.org/jira/browse/FLINK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445667#comment-17445667
]
liuzhuo commented on FLINK-24944:
-
I have an idea to add some functions that can dynamically set parameters on top
of the pod template, such as defining in the pod template:
{code:java}
priorityClassName: ${PriorityClassName}{code}
and then use parameters like `kubernetes.template.env.PriorityClassName` to
dynamically replace the value of `PriorityClassName` of different tasks when
the task is running. so that a unified pod template can be used to run
different Jobs.
> Flink job on k8s mode supports pod scheduling according to priority
> ---
>
> Key: FLINK-24944
> URL: https://issues.apache.org/jira/browse/FLINK-24944
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Reporter: john
>Priority: Major
>
> Hi, is it possible to support the specified pod priority when supporting the
> pod scheduling of the job? I think it is necessary to support the priority
> scheduling of the job in a cluster with multiple users. In Kubernetes 1.8 and
> later versions, the priority of Pod is specified by PriorityClass. At
> present, if I want to specify the priority when submitting a job, I must
> provide a separate pod-templete.yaml for each job, which is often not
> flexible enough. Can we provide a separate parameter similar to
> yarn.application.priority to support the priority of the job when we submit
> the job to k8s,，such as kubernetes .application.priority-class。
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24897) Enable application mode on YARN to use usrlib
[
https://issues.apache.org/jira/browse/FLINK-24897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445664#comment-17445664
]
Yang Wang commented on FLINK-24897:
---
Thanks for addressing the comments.
# The {{usrlib}} should be shipped automatically. We could interrupt with
exception if the user also specified a shipped {{{}usrlib{}}}. This behavior
makes sense to me.
# The {{usrlib}} should also work for per-job mode.
# Both for per-job and application mode, only when UserJarInclusion is
configured to {{{}DISABLED{}}}, the {{usrlib}} will be shipped only and loaded
by user classloader. Otherwise, it should work as user jars, which means all
the jars in {{usrlib}} will be put into {{CUR_DIR}} and added to system
classpath.
> Enable application mode on YARN to use usrlib
> -
>
> Key: FLINK-24897
> URL: https://issues.apache.org/jira/browse/FLINK-24897
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Biao Geng
>Priority: Major
>
> Hi there,
> I am working to utilize application mode to submit flink jobs to YARN cluster
> but I find that currently there is no easy way to ship my user-defined
> jars(e.g. some custom connectors or udf jars that would be shared by some
> jobs) and ask the FlinkUserCodeClassLoader to load classes in these jars.
> I checked some relevant jiras, like FLINK-21289. In k8s mode, there is a
> solution that users can use `usrlib` directory to store their user-defined
> jars and these jars would be loaded by FlinkUserCodeClassLoader when the job
> is executed on JM/TM.
> But on YARN mode, `usrlib` does not work as that:
> In this method(org.apache.flink.yarn.YarnClusterDescriptor#addShipFiles), if
> I want to use `yarn.ship-files` to ship `usrlib` from my flink client(in my
> local machine) to remote cluster, I must not set UserJarInclusion to
> DISABLED due to the checkArgument(). However, if I do not set that option to
> DISABLED, the user jars to be shipped will be added into systemClassPaths. As
> a result, classes in those user jars will be loaded by AppClassLoader.
> But if I do not ship these jars, there is no convenient way to utilize these
> jars in my flink run command. Currently, all I can do seems to use `-C`
> option, which means I have to upload my jars to some shared store first and
> then use these remote paths. It is not so perfect as we have already make it
> possible to ship jars or files directly and we also introduce `usrlib` in
> application mode on YARN. It would be more user-friendly if we can allow
> shipping `usrlib` from local to remote cluster while using
> FlinkUserCodeClassLoader to load classes in the jars in `usrlib`.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17823: [FLINK-24937][e2e] Return correct exit code in build_image
flinkbot edited a comment on pull request #17823: URL: https://github.com/apache/flink/pull/17823#issuecomment-972474977 ## CI report: * fc2c92648891940ca961617c5a7715d3e8137689 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26686) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24924) TO_TIMESTAMP and TO_DATE should fail
[
https://issues.apache.org/jira/browse/FLINK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445660#comment-17445660
]
Jark Wu commented on FLINK-24924:
-
> I agree that exceptions are not optimal for streaming pipelines. But even for
> batch jobs, this is a reoccurring problem. Take a batch job for example that
> runs every night and suddenly produces an exception due to some corrupt data
> in the current batch. It is basically the same problem.
I don't think so. It's very easy for batch jobs to update SQL and re-run.
However, this is super hard for streaming jobs, because the state may not
compatible and the source may expired the needed data.
"COALESCE(CAST('aa' AS INT), 0) => NULL" is a bug of return type of CAST, we
should fix the result type signatures of all functions which returning null.
I think it's not a bad case or hard to explain for returning null.
> TO_TIMESTAMP and TO_DATE should fail
>
>
> Key: FLINK-24924
> URL: https://issues.apache.org/jira/browse/FLINK-24924
> Project: Flink
> Issue Type: Sub-task
>Reporter: Francesco Guardiani
>Priority: Major
>
> In a similar fashion to what described
> https://issues.apache.org/jira/browse/FLINK-24385, TO_TIMESTAMP and TO_DATE
> should fail instead of returning {{null}}.
> In particular for these two functions, a failure in parsing could lead to
> very unexpected behavior, for example it could lead to records with null
> rowtime.
> We should change these functions to fail by default when parsing generates an
> error. We can let users handle errors by letting them use TRY_CAST for the
> same functionality:
> {code:sql}
> -- This fails when input is invalid
> TO_TIMESTAMP(input)
> -- Behaves the same as above
> CAST(input AS TIMESTAMP)
> -- This returns null when input is invalid
> TRY_CAST(input AS TIMESTAMP)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24944) Flink job on k8s mode supports pod scheduling according to priority
[ https://issues.apache.org/jira/browse/FLINK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445657#comment-17445657 ] john commented on FLINK-24944: -- [~wangyang0918] [~guoyangze] Thank you for your review. > Flink job on k8s mode supports pod scheduling according to priority > --- > > Key: FLINK-24944 > URL: https://issues.apache.org/jira/browse/FLINK-24944 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: john >Priority: Major > > Hi, is it possible to support the specified pod priority when supporting the > pod scheduling of the job? I think it is necessary to support the priority > scheduling of the job in a cluster with multiple users. In Kubernetes 1.8 and > later versions, the priority of Pod is specified by PriorityClass. At > present, if I want to specify the priority when submitting a job, I must > provide a separate pod-templete.yaml for each job, which is often not > flexible enough. Can we provide a separate parameter similar to > yarn.application.priority to support the priority of the job when we submit > the job to k8s,，such as kubernetes .application.priority-class。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24942) Could not find any factory for identifier 'hive' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
[
https://issues.apache.org/jira/browse/FLINK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445653#comment-17445653
]
JasonLee commented on FLINK-24942:
--
[~fsk119] After looking at the relevant code, I found that the class
hivedynamictablefactory was not added to meta-inf / services And I tried adding
jar packages with -j but it didn't work
> Could not find any factory for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
>
>
> Key: FLINK-24942
> URL: https://issues.apache.org/jira/browse/FLINK-24942
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
> Environment: Flink-1.14.0
>Reporter: JasonLee
>Priority: Major
> Fix For: 1.15.0
>
>
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Could not find any factory
> for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath.
> Available factory identifiers are:
> blackhole
> datagen
> filesystem
> kafka
> print
> upsert-kafka
>
> The above exception is thrown when I execute the following SQL, even though I
> have added flink-sql-connector-hive-2.3.6_2.11-1.14.0.jar in flink/lib
> {code:java}
> // code placeholder
> insert into fs_table
> select xxx,
> xxx,
> xxx,
> xxx,
> xxx,
> DATE_FORMAT(ts_ltz, '-MM-dd'), DATE_FORMAT(ts_ltz, 'HH')
> from kafka_table; {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24: URL: https://github.com/apache/flink-ml/pull/24#discussion_r751928196 ## File path: pom.xml ## @@ -56,6 +56,7 @@ under the License. flink-ml-iteration flink-ml-lib flink-ml-tests +flink-ml-lib Review comment: OK, I will do it later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24:
URL: https://github.com/apache/flink-ml/pull/24#discussion_r751927736
##
File path:
flink-ml-lib/src/test/java/org/apache/flink/ml/classification/knn/KnnTest.java
##
@@ -0,0 +1,168 @@
+package org.apache.flink.ml.classification.knn;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.api.core.Pipeline;
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+/** knn algorithm test. */
+public class KnnTest {
Review comment:
OK, I will refine the ut later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17824: fix testSimpleNFA
flinkbot edited a comment on pull request #17824: URL: https://github.com/apache/flink/pull/17824#issuecomment-972561439 ## CI report: * 7e2695a4d3ec15a6fb0bdbc55cf2c92794f3aee9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26687) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24:
URL: https://github.com/apache/flink-ml/pull/24#discussion_r751927174
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/distance/EuclideanDistance.java
##
@@ -0,0 +1,259 @@
+package org.apache.flink.ml.classification.knn.distance;
+
+import org.apache.flink.ml.common.linalg.BLAS;
+import org.apache.flink.ml.common.linalg.DenseMatrix;
+import org.apache.flink.ml.common.linalg.DenseVector;
+import org.apache.flink.ml.common.linalg.MatVecOp;
+import org.apache.flink.ml.common.linalg.SparseVector;
+import org.apache.flink.ml.common.linalg.Vector;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Arrays;
+
+/**
+ * Euclidean distance is the "ordinary" straight-line distance between two
points in Euclidean
+ * space.
+ *
+ * https://en.wikipedia.org/wiki/Euclidean_distance
+ *
+ * Given two vectors a and b, Euclidean Distance = ||a - b||, where ||*||
means the L2 norm of
+ * the vector.
+ */
+public class EuclideanDistance extends BaseFastDistance {
Review comment:
Here, the "infra" is not Flink ML infra, it's only the infra of distance
design. It's an inner infra of knn algorithm.
By this infra, algorithm developer can add new distance type, or add new
accelerated method of distance calculation.
I think, this infra doesn't need discussion any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24:
URL: https://github.com/apache/flink-ml/pull/24#discussion_r751927174
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/distance/EuclideanDistance.java
##
@@ -0,0 +1,259 @@
+package org.apache.flink.ml.classification.knn.distance;
+
+import org.apache.flink.ml.common.linalg.BLAS;
+import org.apache.flink.ml.common.linalg.DenseMatrix;
+import org.apache.flink.ml.common.linalg.DenseVector;
+import org.apache.flink.ml.common.linalg.MatVecOp;
+import org.apache.flink.ml.common.linalg.SparseVector;
+import org.apache.flink.ml.common.linalg.Vector;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Arrays;
+
+/**
+ * Euclidean distance is the "ordinary" straight-line distance between two
points in Euclidean
+ * space.
+ *
+ * https://en.wikipedia.org/wiki/Euclidean_distance
+ *
+ * Given two vectors a and b, Euclidean Distance = ||a - b||, where ||*||
means the L2 norm of
+ * the vector.
+ */
+public class EuclideanDistance extends BaseFastDistance {
Review comment:
Here, the "infra" is not Flink ML infra, it's only the infra of distance
design. It's an inner infra of knn algorithm.
By this infra, algorithm developer can add new distance type, or add new
accelerated method of distance calculation.
I think, this infra doesn't need discuss any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17824: fix testSimpleNFA
flinkbot commented on pull request #17824: URL: https://github.com/apache/flink/pull/17824#issuecomment-972561996 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7e2695a4d3ec15a6fb0bdbc55cf2c92794f3aee9 (Thu Nov 18 05:56:48 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17824: fix testSimpleNFA
flinkbot commented on pull request #17824: URL: https://github.com/apache/flink/pull/17824#issuecomment-972561439 ## CI report: * 7e2695a4d3ec15a6fb0bdbc55cf2c92794f3aee9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] arianacai1997 opened a new pull request #17824: fix testSimpleNFA
arianacai1997 opened a new pull request #17824: URL: https://github.com/apache/flink/pull/17824 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* The pull request makes the testSimpleNFA non-flaky by not comparing a list with a collection. ## Brief change log - Change the type of `expectedPatterns` from List to Collection, which is consistent with the `actualPatterns`. - Change the `assertEquals` to `assetTrue` to prepare for comparing two `Collection`. - Use `equals` to compare. ## Verifying this change *(Please pick either of the following options)* This change is already covered by existing tests, such as *(please describe tests)*. By running the nondex commands before and after fixing the flaky test, I see the order will not matter. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes: HashSet, assertTrue) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (don't know) - The runtime per-record code paths (performance sensitive): ( don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (don't know) - The S3 file system connector: ( no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24: URL: https://github.com/apache/flink-ml/pull/24#discussion_r751924593 ## File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/KnnParams.java ## @@ -0,0 +1,38 @@ +package org.apache.flink.ml.classification.knn; + +import org.apache.flink.ml.param.IntParam; +import org.apache.flink.ml.param.Param; +import org.apache.flink.ml.param.ParamValidators; +import org.apache.flink.ml.param.WithParams; +import org.apache.flink.ml.params.shared.colname.HasFeatureColsDefaultAsNull; +import org.apache.flink.ml.params.shared.colname.HasKnnDistanceType; +import org.apache.flink.ml.params.shared.colname.HasLabelCol; +import org.apache.flink.ml.params.shared.colname.HasPredictionCol; +import org.apache.flink.ml.params.shared.colname.HasPredictionDetailCol; +import org.apache.flink.ml.params.shared.colname.HasReservedColsDefaultAsNull; +import org.apache.flink.ml.params.shared.colname.HasVectorColDefaultAsNull; + +/** knn fit parameters. */ +public interface KnnParams Review comment: use one params is OK. I will refine code later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r751924452
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesModel.java
##
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** A Model which classifies data using the model data computed by {@link
NaiveBayes}. */
+public class NaiveBayesModel
+implements Model,
NaiveBayesModelParams {
+private static final long serialVersionUID = -4673084154965905629L;
+private final Map, Object> paramMap = new HashMap<>();
+private Table modelTable;
+private static final String broadcastModelKey = "NaiveBayesModelStream";
Review comment:
`broadcastModelKey` is also used when creating broadcastMap, but it's
true that this variable is only used in one public method and its private
methods. I'll make `broadcastModelKey` a local variable of `transform()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24: URL: https://github.com/apache/flink-ml/pull/24#discussion_r751923670 ## File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/KnnParams.java ## @@ -0,0 +1,38 @@ +package org.apache.flink.ml.classification.knn; + +import org.apache.flink.ml.param.IntParam; +import org.apache.flink.ml.param.Param; +import org.apache.flink.ml.param.ParamValidators; +import org.apache.flink.ml.param.WithParams; +import org.apache.flink.ml.params.shared.colname.HasFeatureColsDefaultAsNull; +import org.apache.flink.ml.params.shared.colname.HasKnnDistanceType; +import org.apache.flink.ml.params.shared.colname.HasLabelCol; +import org.apache.flink.ml.params.shared.colname.HasPredictionCol; +import org.apache.flink.ml.params.shared.colname.HasPredictionDetailCol; +import org.apache.flink.ml.params.shared.colname.HasReservedColsDefaultAsNull; +import org.apache.flink.ml.params.shared.colname.HasVectorColDefaultAsNull; + +/** knn fit parameters. */ +public interface KnnParams +extends WithParams, +HasVectorColDefaultAsNull, +HasKnnDistanceType, +HasLabelCol, +HasFeatureColsDefaultAsNull, +HasPredictionCol, +HasPredictionDetailCol, Review comment: If user want to get two or three possible classification results, by setting HasDetail knn will give more classification results. then user can do more thing with the detail, just like recommendation. if user want to give only one classification result, he need not set this parameter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24: URL: https://github.com/apache/flink-ml/pull/24#discussion_r751921706 ## File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/KnnParams.java ## @@ -0,0 +1,38 @@ +package org.apache.flink.ml.classification.knn; + +import org.apache.flink.ml.param.IntParam; +import org.apache.flink.ml.param.Param; +import org.apache.flink.ml.param.ParamValidators; +import org.apache.flink.ml.param.WithParams; +import org.apache.flink.ml.params.shared.colname.HasFeatureColsDefaultAsNull; +import org.apache.flink.ml.params.shared.colname.HasKnnDistanceType; +import org.apache.flink.ml.params.shared.colname.HasLabelCol; +import org.apache.flink.ml.params.shared.colname.HasPredictionCol; +import org.apache.flink.ml.params.shared.colname.HasPredictionDetailCol; +import org.apache.flink.ml.params.shared.colname.HasReservedColsDefaultAsNull; +import org.apache.flink.ml.params.shared.colname.HasVectorColDefaultAsNull; + +/** knn fit parameters. */ +public interface KnnParams +extends WithParams, +HasVectorColDefaultAsNull, +HasKnnDistanceType, Review comment: I don't think Measure is better than type. DistanceType is more appropriate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a change in pull request #24: Flink 24557
weibozhao commented on a change in pull request #24:
URL: https://github.com/apache/flink-ml/pull/24#discussion_r751920637
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/BatchOperator.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.table.api.Table;
+
+import java.util.Arrays;
+import java.util.Map;
+
+/**
+ * Base class of offline learning algorithm operators.
+ *
+ * This class extends {@link BaseAlgoImpl} to support data transmission
between BatchOperators.
+ */
+public abstract class BatchOperator> extends
BaseAlgoImpl {
Review comment:
I will remove them later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17811: [FLINK-24754] Implement CHAR/VARCHAR length validation for sinks
flinkbot edited a comment on pull request #17811: URL: https://github.com/apache/flink/pull/17811#issuecomment-970441907 ## CI report: * 2b885bb3d72eb7489b8aa3211d63a7c1b7ca591d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26682) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17759: [FLINK-24608][table-planner][table-runtime] Fix issue with missing timestamp
flinkbot edited a comment on pull request #17759: URL: https://github.com/apache/flink/pull/17759#issuecomment-965566767 ## CI report: * 61b8df758969cd93abf497205f4a43c224176efb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26681) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24897) Enable application mode on YARN to use usrlib
[
https://issues.apache.org/jira/browse/FLINK-24897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445631#comment-17445631
]
Biao Geng commented on FLINK-24897:
---
Hi [~trohrmann] and [~wangyang0918] thank you very much for your reply.
I agree with Till's suggestion about reusing the existing logic to include
{{usrlib}} in user classloader.
Yang's questions are also helpful and critical:
*A summary of my answer abourt {{{}usrlib{}}}:*
0. We should ship {{usrlib}} by default like what we have done for {{lib}} dir.
1. We should avoid uploading it again and not add classes in it into system
path if users specify {{usrlib}} again in the {{yarn.ship-files}} option.
2. It should work for per-job mode
3. Only when UserJarInclusion is DISABLED will {{usrlib}} take effect in
per-job mode. But we should consider the default value of {{UserJarInclusion}}
option.
*Datail:*
Q1:
Currently, I think we should ship {{usrlib}} by default if it exists because
AFAIK, {{usrlib}} is the default userClassPath which is defined by flink. If we
ask the user to explicitly specify it, it is somehow waste the flink's contract
with users.
When users specify a shipped directory named as "usrlib", I think there are 3
options:
Option1: skip it
Option2: report error
Option3: do nothing but just upload it and add files in {{usrlib}} into system
classpaths
Option1 seems to be easiest, just as what we have done for {{flink_dist.jar}}
when users specify {{lib}} in ship files.
Option3 is worthwhile to mention as if users specify {{usrlib}} in ship files,
files in {{usrlib}} will be added into system classpaths but if users use
child-first resolve order, files in {{usrlib}} will also be loaded by
UserClassLoader as they are in userClassPath as well. Bad things happen If
users choose parent-first resolve order, files in {{usrlib}} will be loaded by
AppClassLoader which breaks the design.
So, in summary, I think skipping it is a better one.
Q2:
After checking codes about {{FileJobGraphRetriever}} and
{{{}YarnJobClusterEntrypoint{}}}, I think we have prepared for using {{usrlib}}
if we upload it to the cluster.
Q3:
I agree only when UserJarInclusion is DISABLED will {{usrlib}} take effect in
per-job mode. But currently default value of UserJarInclusion is {{ORDERED}}
and works for all 3 modes(per job, session, app). If we agree the {{usrlib}}
should be shipped automatically, we may need to consider the default value of
this option if we want to use UserClassLoader to load jars in {{{}usrlib{}}}.
> Enable application mode on YARN to use usrlib
> -
>
> Key: FLINK-24897
> URL: https://issues.apache.org/jira/browse/FLINK-24897
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Biao Geng
>Priority: Major
>
> Hi there,
> I am working to utilize application mode to submit flink jobs to YARN cluster
> but I find that currently there is no easy way to ship my user-defined
> jars(e.g. some custom connectors or udf jars that would be shared by some
> jobs) and ask the FlinkUserCodeClassLoader to load classes in these jars.
> I checked some relevant jiras, like FLINK-21289. In k8s mode, there is a
> solution that users can use `usrlib` directory to store their user-defined
> jars and these jars would be loaded by FlinkUserCodeClassLoader when the job
> is executed on JM/TM.
> But on YARN mode, `usrlib` does not work as that:
> In this method(org.apache.flink.yarn.YarnClusterDescriptor#addShipFiles), if
> I want to use `yarn.ship-files` to ship `usrlib` from my flink client(in my
> local machine) to remote cluster, I must not set UserJarInclusion to
> DISABLED due to the checkArgument(). However, if I do not set that option to
> DISABLED, the user jars to be shipped will be added into systemClassPaths. As
> a result, classes in those user jars will be loaded by AppClassLoader.
> But if I do not ship these jars, there is no convenient way to utilize these
> jars in my flink run command. Currently, all I can do seems to use `-C`
> option, which means I have to upload my jars to some shared store first and
> then use these remote paths. It is not so perfect as we have already make it
> possible to ship jars or files directly and we also introduce `usrlib` in
> application mode on YARN. It would be more user-friendly if we can allow
> shipping `usrlib` from local to remote cluster while using
> FlinkUserCodeClassLoader to load classes in the jars in `usrlib`.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24941) Cannot report backpressure with DatadogReporter
[
https://issues.apache.org/jira/browse/FLINK-24941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445628#comment-17445628
]
Shen Zhu commented on FLINK-24941:
--
+1. I also encountered this problem.
> Cannot report backpressure with DatadogReporter
> ---
>
> Key: FLINK-24941
> URL: https://issues.apache.org/jira/browse/FLINK-24941
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Ori Popowski
>Priority: Major
>
> When using {{DatadogHttpReporter}} the log is full of these errors:
>
> {code:java}
> 2021-11-16 09:51:11,521 [Flink-MetricRegistry-thread-1] INFO
> org.apache.flink.metrics.datadog.DatadogHttpReporter - The metric
> flink.task.isBackPressured will not be reported because only number types are
> supported by this reporter. {code}
> The code shows that the reason is that {{isBackPressured}} is a Boolean and
> all Gauge values are converted to {{Number}} which results in
> {{ClassCastException}} [1].
> I understand the limitation, but:
> # This bug can be easily fixed
> # Monitoring backpressure is extremely important. Without backpressure
> monitroing there's no way of seeing backpressure history and no alerts.
> h3. Workaround
> For anyone interested, rewrite the
> {{org.apache.flink.metrics.datadog.DGauge}} to map Booleans to integers (0 =>
> false, 1 => true), and use the maven/sbt shade plugin to take your own
> version of this class into the final JAR instead the existing class from the
> flink-metrics-datadog package.
>
> [1]
> https://github.com/apache/flink/blob/release-1.11/flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java#L184-L188
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] lindong28 commented on a change in pull request #24: Flink 24557
lindong28 commented on a change in pull request #24:
URL: https://github.com/apache/flink-ml/pull/24#discussion_r751889202
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/KnnParams.java
##
@@ -0,0 +1,38 @@
+package org.apache.flink.ml.classification.knn;
+
+import org.apache.flink.ml.param.IntParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.WithParams;
+import org.apache.flink.ml.params.shared.colname.HasFeatureColsDefaultAsNull;
+import org.apache.flink.ml.params.shared.colname.HasKnnDistanceType;
+import org.apache.flink.ml.params.shared.colname.HasLabelCol;
+import org.apache.flink.ml.params.shared.colname.HasPredictionCol;
+import org.apache.flink.ml.params.shared.colname.HasPredictionDetailCol;
+import org.apache.flink.ml.params.shared.colname.HasReservedColsDefaultAsNull;
+import org.apache.flink.ml.params.shared.colname.HasVectorColDefaultAsNull;
+
+/** knn fit parameters. */
+public interface KnnParams
Review comment:
Does `Knn` and `KnnModel` share exactly the same set of parameters? If
not, we have separate parameter classes for them, e.g. `KnnParams` and
`KnnModelParams`?
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/KnnParams.java
##
@@ -0,0 +1,38 @@
+package org.apache.flink.ml.classification.knn;
+
+import org.apache.flink.ml.param.IntParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.WithParams;
+import org.apache.flink.ml.params.shared.colname.HasFeatureColsDefaultAsNull;
+import org.apache.flink.ml.params.shared.colname.HasKnnDistanceType;
+import org.apache.flink.ml.params.shared.colname.HasLabelCol;
+import org.apache.flink.ml.params.shared.colname.HasPredictionCol;
+import org.apache.flink.ml.params.shared.colname.HasPredictionDetailCol;
+import org.apache.flink.ml.params.shared.colname.HasReservedColsDefaultAsNull;
+import org.apache.flink.ml.params.shared.colname.HasVectorColDefaultAsNull;
+
+/** knn fit parameters. */
+public interface KnnParams
+extends WithParams,
+HasVectorColDefaultAsNull,
+HasKnnDistanceType,
+HasLabelCol,
+HasFeatureColsDefaultAsNull,
+HasPredictionCol,
+HasPredictionDetailCol,
Review comment:
Spark does not support `HasPredictionDetailCol`. Any chance we can try
to be consistent with Spark user experience for now? Otherwise, could you help
explain the use-case and explain how this use-case is supported for Spark users?
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/knn/distance/EuclideanDistance.java
##
@@ -0,0 +1,259 @@
+package org.apache.flink.ml.classification.knn.distance;
+
+import org.apache.flink.ml.common.linalg.BLAS;
+import org.apache.flink.ml.common.linalg.DenseMatrix;
+import org.apache.flink.ml.common.linalg.DenseVector;
+import org.apache.flink.ml.common.linalg.MatVecOp;
+import org.apache.flink.ml.common.linalg.SparseVector;
+import org.apache.flink.ml.common.linalg.Vector;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Arrays;
+
+/**
+ * Euclidean distance is the "ordinary" straight-line distance between two
points in Euclidean
+ * space.
+ *
+ * https://en.wikipedia.org/wiki/Euclidean_distance
+ *
+ * Given two vectors a and b, Euclidean Distance = ||a - b||, where ||*||
means the L2 norm of
+ * the vector.
+ */
+public class EuclideanDistance extends BaseFastDistance {
Review comment:
Can we re-use `EuclideanDistanceMeasure` for now? Otherwise, we will
need to discuss the design for Flink ML infra classes such as
`FastDistanceVectorData`, which will take sometime I guess.
Same for other Flink ML infra classes.
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/linalg/SparseVector.java
##
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.linalg;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import
[jira] [Commented] (FLINK-24942) Could not find any factory for identifier 'hive' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
[
https://issues.apache.org/jira/browse/FLINK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445624#comment-17445624
]
Shengkai Fang commented on FLINK-24942:
---
It seems the sql-client doesn't find the hive jar. You can start the sql client
with -j option to specify which jar should be loaded.
> Could not find any factory for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
>
>
> Key: FLINK-24942
> URL: https://issues.apache.org/jira/browse/FLINK-24942
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
> Environment: Flink-1.14.0
>Reporter: JasonLee
>Priority: Major
> Fix For: 1.15.0
>
>
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Could not find any factory
> for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath.
> Available factory identifiers are:
> blackhole
> datagen
> filesystem
> kafka
> print
> upsert-kafka
>
> The above exception is thrown when I execute the following SQL, even though I
> have added flink-sql-connector-hive-2.3.6_2.11-1.14.0.jar in flink/lib
> {code:java}
> // code placeholder
> insert into fs_table
> select xxx,
> xxx,
> xxx,
> xxx,
> xxx,
> DATE_FORMAT(ts_ltz, '-MM-dd'), DATE_FORMAT(ts_ltz, 'HH')
> from kafka_table; {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24942) Could not find any factory for identifier 'hive' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
[
https://issues.apache.org/jira/browse/FLINK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445621#comment-17445621
]
JasonLee commented on FLINK-24942:
--
[~ruanhang1993]
SET table.sql-dialect=hive;
drop table if exists fs_table;
CREATE TABLE fs_table (
xxx STRING,
xxx INT,
xxx STRING,
xxx STRING,
xxx BIGINT
) PARTITIONED BY (dt STRING,`hour` STRING) STORED AS PARQUET TBLPROPERTIES (
'sink.partition-commit.delay'='1s',
'sink.partition-commit.policy.kind'='metastore,success-file',
'sink.rolling-policy.check-interval'='1min',
'sink.partition-commit.trigger'='partition-time',
'partition.time-extractor.timestamp-pattern'='$dt $hr:00:00',
'sink.partition-commit.watermark-time-zone'='Asia/Shanghai'
);
> Could not find any factory for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
>
>
> Key: FLINK-24942
> URL: https://issues.apache.org/jira/browse/FLINK-24942
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
> Environment: Flink-1.14.0
>Reporter: JasonLee
>Priority: Major
> Fix For: 1.15.0
>
>
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Could not find any factory
> for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath.
> Available factory identifiers are:
> blackhole
> datagen
> filesystem
> kafka
> print
> upsert-kafka
>
> The above exception is thrown when I execute the following SQL, even though I
> have added flink-sql-connector-hive-2.3.6_2.11-1.14.0.jar in flink/lib
> {code:java}
> // code placeholder
> insert into fs_table
> select xxx,
> xxx,
> xxx,
> xxx,
> xxx,
> DATE_FORMAT(ts_ltz, '-MM-dd'), DATE_FORMAT(ts_ltz, 'HH')
> from kafka_table; {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] lindong28 edited a comment on pull request #24: Flink 24557
lindong28 edited a comment on pull request #24: URL: https://github.com/apache/flink-ml/pull/24#issuecomment-972499228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #24: Flink 24557
lindong28 commented on pull request #24: URL: https://github.com/apache/flink-ml/pull/24#issuecomment-972499228 Thanks for the update @weibozhao. Here are some PR review convention that might be useful to make the review easier: - Maybe change the PR title and the commit message to follow the style of other Flink PRs. For example, `[FLINK-24557] Support Knn in Flink ML`. Even better, maybe we can use the same message as other algorithm PRs, e.g. `[FLINK-24557] Add KNN classification algorithm`. -Can you resolve the comments that you have addressed on this Github page? This allows us to see which comments need further discussion? - Can you clean up commit message in such a way that those commits can be merged as is? The PR currently contains commit messages such as `refine knn code format` which is probably not necessary. Please feel free to follow commit messages in the existing Flink repository for example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19358) when submit job on application mode with HA,the jobid will be 0000000000
[
https://issues.apache.org/jira/browse/FLINK-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445618#comment-17445618
]
Yang Wang commented on FLINK-19358:
---
For single-execute application, relying on the {{JobGraphStore}} and having a
random job id is reasonable. Since we do not have a concrete plan to support
multiple-execute application with HA mode, maybe we could consider this
solution.
> when submit job on application mode with HA,the jobid will be 00
>
>
> Key: FLINK-19358
> URL: https://issues.apache.org/jira/browse/FLINK-19358
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Jun Zhang
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> when submit a flink job on application mode with HA ,the flink job id will be
> , when I have many jobs ,they have the same
> job id , it will be lead to a checkpoint error
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r751884914
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+implements Estimator,
NaiveBayesParams {
+private final Map, Object> paramMap = new HashMap<>();
+
+public NaiveBayes() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public NaiveBayesModel fit(Table... inputs) {
+final String featuresCol = getFeaturesCol();
Review comment:
OK. I'll add this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r751875906
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * An Estimator which implements the naive bayes classification algorithm.
+ *
+ * See https://en.wikipedia.org/wiki/Naive_Bayes_classifier.
+ */
+public class NaiveBayes
+implements Estimator,
NaiveBayesParams {
+private final Map, Object> paramMap = new HashMap<>();
+
+public NaiveBayes() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public NaiveBayesModel fit(Table... inputs) {
+final String featuresCol = getFeaturesCol();
Review comment:
I realized that it is probably better to setup the best practice of
always using `Preconditions.checkArgument(...)` to verify the number of inputs
is expected, for every public APIs such as fit/transform/setModelData.
Could you help add this check for both NaiveBayes and KMeans?
##
File path:
flink-ml-lib/src/test/java/org/apache/flink/ml/classification/NaiveBayesTest.java
##
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayes;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModel;
+import org.apache.flink.ml.classification.naivebayes.NaiveBayesModelData;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import
org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
[jira] [Commented] (FLINK-24944) Flink job on k8s mode supports pod scheduling according to priority
[ https://issues.apache.org/jira/browse/FLINK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445615#comment-17445615 ] Yang Wang commented on FLINK-24944: --- I also believe that using the pod template is enough. Actually, we could not support all the K8s features(e.g. priority, schedulerName, preemptionPolicy, dnsPolicy, etc.) via Flink config options. And the pod template is designed to support those advanced features. > Flink job on k8s mode supports pod scheduling according to priority > --- > > Key: FLINK-24944 > URL: https://issues.apache.org/jira/browse/FLINK-24944 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: john >Priority: Major > > Hi, is it possible to support the specified pod priority when supporting the > pod scheduling of the job? I think it is necessary to support the priority > scheduling of the job in a cluster with multiple users. In Kubernetes 1.8 and > later versions, the priority of Pod is specified by PriorityClass. At > present, if I want to specify the priority when submitting a job, I must > provide a separate pod-templete.yaml for each job, which is often not > flexible enough. Can we provide a separate parameter similar to > yarn.application.priority to support the priority of the job when we submit > the job to k8s,，such as kubernetes .application.priority-class。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wangyang0918 commented on pull request #17823: [FLINK-24937][e2e] Return correct exit code in build_image
wangyang0918 commented on pull request #17823: URL: https://github.com/apache/flink/pull/17823#issuecomment-972476539 cc @zentol would you please have a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17823: [FLINK-24937][e2e] Return correct exit code in build_image
flinkbot edited a comment on pull request #17823: URL: https://github.com/apache/flink/pull/17823#issuecomment-972474977 ## CI report: * fc2c92648891940ca961617c5a7715d3e8137689 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26686) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17759: [FLINK-24608][table-planner][table-runtime] Fix issue with missing timestamp
flinkbot edited a comment on pull request #17759: URL: https://github.com/apache/flink/pull/17759#issuecomment-965566767 ## CI report: * e6d866bebbfeb98c26e06dc831feb9e21db7ca3e Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26680) * 61b8df758969cd93abf497205f4a43c224176efb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26681) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17823: [FLINK-24937][e2e] Return correct exit code in build_image
flinkbot commented on pull request #17823: URL: https://github.com/apache/flink/pull/17823#issuecomment-972475817 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit fc2c92648891940ca961617c5a7715d3e8137689 (Thu Nov 18 03:10:20 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17823: [FLINK-24937][e2e] Return correct exit code in build_image
flinkbot commented on pull request #17823: URL: https://github.com/apache/flink/pull/17823#issuecomment-972474977 ## CI report: * fc2c92648891940ca961617c5a7715d3e8137689 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24937) "kubernetes application HA test" hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-24937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24937:
---
Labels: pull-request-available test-stability (was: test-stability)
> "kubernetes application HA test" hangs on azure
> ---
>
> Key: FLINK-24937
> URL: https://issues.apache.org/jira/browse/FLINK-24937
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.1
>Reporter: Yun Gao
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {code:java}
> 2021-11-17T02:15:23.4319085Z Nov 17 02:15:23 ##[endgroup]
> 2021-11-17T02:15:23.4319779Z Nov 17 02:15:23 Searching for .dump, .dumpstream
> and related files in '/home/vsts/work/1/s'
> 2021-11-17T02:15:24.7906640Z dmesg: read kernel buffer failed: Operation not
> permitted
> 2021-11-17T02:15:25.1156830Z Nov 17 02:15:25 No taskexecutor daemon to stop
> on host fv-az123-364.
> 2021-11-17T02:15:25.3229454Z Nov 17 02:15:25 No standalonesession daemon to
> stop on host fv-az123-364.
> 2021-11-17T06:09:57.2488148Z
> ==
> 2021-11-17T06:09:57.2489380Z === WARNING: This task took already 95% of the
> available time budget of 281 minutes ===
> 2021-11-17T06:09:57.2490646Z
> ==
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26628=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=2562
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] wangyang0918 opened a new pull request #17823: [FLINK-24937][e2e] Return correct exit code in build_image
wangyang0918 opened a new pull request #17823: URL: https://github.com/apache/flink/pull/17823 ## What is the purpose of the change Let the `build_image` in `common_docker.sh` return correct exit code. This will make `retry_times build_image` could work normally. ## Brief change log * [FLINK-24937][e2e] Return correct exit code in build_image ## Verifying this change * All the existing tests should pass ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24937) "kubernetes application HA test" hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-24937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445610#comment-17445610
]
Yun Gao commented on FLINK-24937:
-
Very thanks [~wangyang0918] for investigating and attending this issue!
> "kubernetes application HA test" hangs on azure
> ---
>
> Key: FLINK-24937
> URL: https://issues.apache.org/jira/browse/FLINK-24937
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.1
>Reporter: Yun Gao
>Assignee: Yang Wang
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {code:java}
> 2021-11-17T02:15:23.4319085Z Nov 17 02:15:23 ##[endgroup]
> 2021-11-17T02:15:23.4319779Z Nov 17 02:15:23 Searching for .dump, .dumpstream
> and related files in '/home/vsts/work/1/s'
> 2021-11-17T02:15:24.7906640Z dmesg: read kernel buffer failed: Operation not
> permitted
> 2021-11-17T02:15:25.1156830Z Nov 17 02:15:25 No taskexecutor daemon to stop
> on host fv-az123-364.
> 2021-11-17T02:15:25.3229454Z Nov 17 02:15:25 No standalonesession daemon to
> stop on host fv-az123-364.
> 2021-11-17T06:09:57.2488148Z
> ==
> 2021-11-17T06:09:57.2489380Z === WARNING: This task took already 95% of the
> available time budget of 281 minutes ===
> 2021-11-17T06:09:57.2490646Z
> ==
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26628=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=2562
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24937) "kubernetes application HA test" hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-24937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-24937:
--
Fix Version/s: 1.15.0
1.14.1
1.13.4
> "kubernetes application HA test" hangs on azure
> ---
>
> Key: FLINK-24937
> URL: https://issues.apache.org/jira/browse/FLINK-24937
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.1
>Reporter: Yun Gao
>Assignee: Yang Wang
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {code:java}
> 2021-11-17T02:15:23.4319085Z Nov 17 02:15:23 ##[endgroup]
> 2021-11-17T02:15:23.4319779Z Nov 17 02:15:23 Searching for .dump, .dumpstream
> and related files in '/home/vsts/work/1/s'
> 2021-11-17T02:15:24.7906640Z dmesg: read kernel buffer failed: Operation not
> permitted
> 2021-11-17T02:15:25.1156830Z Nov 17 02:15:25 No taskexecutor daemon to stop
> on host fv-az123-364.
> 2021-11-17T02:15:25.3229454Z Nov 17 02:15:25 No standalonesession daemon to
> stop on host fv-az123-364.
> 2021-11-17T06:09:57.2488148Z
> ==
> 2021-11-17T06:09:57.2489380Z === WARNING: This task took already 95% of the
> available time budget of 281 minutes ===
> 2021-11-17T06:09:57.2490646Z
> ==
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26628=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=2562
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24631) Avoiding directly use the labels as selector for deployment and service
[
https://issues.apache.org/jira/browse/FLINK-24631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-24631:
--
Affects Version/s: 1.13.3
> Avoiding directly use the labels as selector for deployment and service
> ---
>
> Key: FLINK-24631
> URL: https://issues.apache.org/jira/browse/FLINK-24631
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.13.3
>Reporter: Aitozi
>Assignee: Aitozi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> We create deployment use the pod selector directly from labels, which is not
> necessary and may cause problem when some user label value have changed (may
> be by third-party system). This may lead to dangling pod or service can not
> select pods.
> I suggest to use minimal and stable flink internal selectors to select the
> JobManager pod like {{app=xxx, component=jobmanager}} and service,
> taskmanager pod and so on.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-19358) when submit job on application mode with HA,the jobid will be 0000000000
[ https://issues.apache.org/jira/browse/FLINK-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445608#comment-17445608 ] jocean.shi commented on FLINK-19358: it don't have any disadvantages to add a random jobId, it's very convenience for user. The rules of checkpoint、jobHistory、and so on will be the same as before.I don't need to modify my code to resolve the sence that all the job have the same jobId. > when submit job on application mode with HA,the jobid will be 00 > > > Key: FLINK-19358 > URL: https://issues.apache.org/jira/browse/FLINK-19358 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: Jun Zhang >Priority: Minor > Labels: auto-deprioritized-major, usability > > when submit a flink job on application mode with HA ,the flink job id will be > , when I have many jobs ,they have the same > job id , it will be lead to a checkpoint error -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] JingsongLi commented on pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
JingsongLi commented on pull request #17520: URL: https://github.com/apache/flink/pull/17520#issuecomment-972468104 Hi all, I'll give my understanding. (Correct me if I am wrong) ## Object ArrayList vs Lazy deserialization As long as the objects inside the `ArrayList` do not fall into the GC old area, the performance difference is not significant. If we use `ArrayList`. There is a trade-off: - Larger capacity: With the complexity of downstream processing, it may cause elements to fall into the GC full zone. - Smaller capacity: The extreme case is 1, which is too costly for `BlockArrayQueue` and seriously affects throughput. Since this trade-off is more difficult to control, we try not to apply a collection of objects. If we must bundle data, we apply a structure similar to BytesMap (only binary, no objects). ## Lazy deserialization in StreamFormat The key problem is that `StreamFormat` has no way to know the real demarcation point of the implementation, which may cause the implementation to hit an EOF exception. Is it possible for StreamFormat to expose a block-like interface that allows implementations to define the demarcation of a block, or each compressed block defines the demarcation point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17810: [BP-1.13][FLINK-24631][k8s] Use a stable subset of labels to select jobManager and taskManager pods
flinkbot edited a comment on pull request #17810: URL: https://github.com/apache/flink/pull/17810#issuecomment-970348193 ## CI report: * 75185c64d632a7a4ee1e5293773d238781de786e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26643) * 6feab49fae2754d12ad4d40a317cd9eeda8f46f2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26685) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24897) Enable application mode on YARN to use usrlib
[ https://issues.apache.org/jira/browse/FLINK-24897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445605#comment-17445605 ] Biao Geng commented on FLINK-24897: --- [~long jiang] , thanks a lot for your advice. I just focus on the local `usrlib` but your solution about supporting hdfs is also a good point. I am also wondering when you implement the `usrlib` by yourself, will you guys ship it by default or ask the user to specify it. especially when you support directories in hdfs? > Enable application mode on YARN to use usrlib > - > > Key: FLINK-24897 > URL: https://issues.apache.org/jira/browse/FLINK-24897 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Biao Geng >Priority: Major > > Hi there, > I am working to utilize application mode to submit flink jobs to YARN cluster > but I find that currently there is no easy way to ship my user-defined > jars(e.g. some custom connectors or udf jars that would be shared by some > jobs) and ask the FlinkUserCodeClassLoader to load classes in these jars. > I checked some relevant jiras, like FLINK-21289. In k8s mode, there is a > solution that users can use `usrlib` directory to store their user-defined > jars and these jars would be loaded by FlinkUserCodeClassLoader when the job > is executed on JM/TM. > But on YARN mode, `usrlib` does not work as that: > In this method(org.apache.flink.yarn.YarnClusterDescriptor#addShipFiles), if > I want to use `yarn.ship-files` to ship `usrlib` from my flink client(in my > local machine) to remote cluster, I must not set UserJarInclusion to > DISABLED due to the checkArgument(). However, if I do not set that option to > DISABLED, the user jars to be shipped will be added into systemClassPaths. As > a result, classes in those user jars will be loaded by AppClassLoader. > But if I do not ship these jars, there is no convenient way to utilize these > jars in my flink run command. Currently, all I can do seems to use `-C` > option, which means I have to upload my jars to some shared store first and > then use these remote paths. It is not so perfect as we have already make it > possible to ship jars or files directly and we also introduce `usrlib` in > application mode on YARN. It would be more user-friendly if we can allow > shipping `usrlib` from local to remote cluster while using > FlinkUserCodeClassLoader to load classes in the jars in `usrlib`. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-24937) "kubernetes application HA test" hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-24937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang reassigned FLINK-24937:
-
Assignee: Yang Wang
> "kubernetes application HA test" hangs on azure
> ---
>
> Key: FLINK-24937
> URL: https://issues.apache.org/jira/browse/FLINK-24937
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.1
>Reporter: Yun Gao
>Assignee: Yang Wang
>Priority: Major
> Labels: test-stability
>
> {code:java}
> 2021-11-17T02:15:23.4319085Z Nov 17 02:15:23 ##[endgroup]
> 2021-11-17T02:15:23.4319779Z Nov 17 02:15:23 Searching for .dump, .dumpstream
> and related files in '/home/vsts/work/1/s'
> 2021-11-17T02:15:24.7906640Z dmesg: read kernel buffer failed: Operation not
> permitted
> 2021-11-17T02:15:25.1156830Z Nov 17 02:15:25 No taskexecutor daemon to stop
> on host fv-az123-364.
> 2021-11-17T02:15:25.3229454Z Nov 17 02:15:25 No standalonesession daemon to
> stop on host fv-az123-364.
> 2021-11-17T06:09:57.2488148Z
> ==
> 2021-11-17T06:09:57.2489380Z === WARNING: This task took already 95% of the
> available time budget of 281 minutes ===
> 2021-11-17T06:09:57.2490646Z
> ==
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26628=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=2562
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17685: [FLINK-24631][Kubernetes]Use minimal selector to select jobManager and taskManager pod
flinkbot edited a comment on pull request #17685: URL: https://github.com/apache/flink/pull/17685#issuecomment-961148655 ## CI report: * 164b09750d797e4686133ca89af9f84895eab8a2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26644) * 8c1def7a1d20b4440e0d6fdb3b9beeeb200e85f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26683) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17810: [BP-1.13][FLINK-24631][k8s] Use a stable subset of labels to select jobManager and taskManager pods
flinkbot edited a comment on pull request #17810: URL: https://github.com/apache/flink/pull/17810#issuecomment-970348193 ## CI report: * 75185c64d632a7a4ee1e5293773d238781de786e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26643) * 6feab49fae2754d12ad4d40a317cd9eeda8f46f2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17809: [BP-1.14][FLINK-24631][k8s] Use a stable subset of labels to select jobManager and taskManager pods
flinkbot edited a comment on pull request #17809: URL: https://github.com/apache/flink/pull/17809#issuecomment-970332023 ## CI report: * e03bc3af99dca743c20766e18eb3ef94b18e98ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26645) * 6c854a3c03d213703bb94d904e822b2239deef1c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26684) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17809: [BP-1.14][FLINK-24631][k8s] Use a stable subset of labels to select jobManager and taskManager pods
flinkbot edited a comment on pull request #17809: URL: https://github.com/apache/flink/pull/17809#issuecomment-970332023 ## CI report: * e03bc3af99dca743c20766e18eb3ef94b18e98ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26645) * 6c854a3c03d213703bb94d904e822b2239deef1c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17685: [FLINK-24631][Kubernetes]Use minimal selector to select jobManager and taskManager pod
flinkbot edited a comment on pull request #17685: URL: https://github.com/apache/flink/pull/17685#issuecomment-961148655 ## CI report: * 164b09750d797e4686133ca89af9f84895eab8a2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26644) * 8c1def7a1d20b4440e0d6fdb3b9beeeb200e85f8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #17685: [FLINK-24631][Kubernetes]Use minimal selector to select jobManager and taskManager pod
wangyang0918 commented on a change in pull request #17685: URL: https://github.com/apache/flink/pull/17685#discussion_r751862261 ## File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesJobManagerFactory.java ## @@ -120,7 +120,7 @@ private static Deployment createJobManagerDeployment( .withSpec(resolvedPod.getSpec()) .endTemplate() .editOrNewSelector() -.addToMatchLabels(labels) + .addToMatchLabels(kubernetesJobManagerParameters.getSelectors()) Review comment: Useless variable `labels`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24763) ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445602#comment-17445602
]
Jingsong Lee commented on FLINK-24763:
--
[~arvid] Hi, Considering the instability of this case, can we merge this PR
first? I think it is harmless.
If the failure is reproduced later, we can consider ignoring this case first
and look deeply.
> ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
> ---
>
> Key: FLINK-24763
> URL: https://issues.apache.org/jira/browse/FLINK-24763
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Jingsong Lee
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> The test {{ParquetFileSystemITCase.testLimitableBulkFormat}} fails with
> {code}
> 2021-11-03T22:10:11.5106075Z Nov 03 22:10:11 [ERROR]
> testLimitableBulkFormat[false] Time elapsed: 9.177 s <<< ERROR!
> 2021-11-03T22:10:11.5106643Z Nov 03 22:10:11 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-11-03T22:10:11.5107213Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-11-03T22:10:11.5111034Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-11-03T22:10:11.5112190Z Nov 03 22:10:11 at
> org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:188)
> 2021-11-03T22:10:11.5112892Z Nov 03 22:10:11 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2021-11-03T22:10:11.5113393Z Nov 03 22:10:11 at
> org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
> 2021-11-03T22:10:11.5114157Z Nov 03 22:10:11 at
> org.apache.flink.formats.parquet.ParquetFileSystemITCase.testLimitableBulkFormat(ParquetFileSystemITCase.java:128)
> 2021-11-03T22:10:11.5114951Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-11-03T22:10:11.5115568Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-11-03T22:10:11.5116115Z Nov 03 22:10:11 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-11-03T22:10:11.5116591Z Nov 03 22:10:11 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-11-03T22:10:11.5117088Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-11-03T22:10:11.5117807Z Nov 03 22:10:11 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-11-03T22:10:11.5118821Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-11-03T22:10:11.5119417Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-11-03T22:10:11.5119944Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-11-03T22:10:11.5120427Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-11-03T22:10:11.5120919Z Nov 03 22:10:11 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-11-03T22:10:11.5121571Z Nov 03 22:10:11 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-11-03T22:10:11.5122526Z Nov 03 22:10:11 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-03T22:10:11.5123245Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-11-03T22:10:11.5123804Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-11-03T22:10:11.5124314Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-11-03T22:10:11.5124806Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-11-03T22:10:11.5125313Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-11-03T22:10:11.5125810Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-11-03T22:10:11.5126281Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-11-03T22:10:11.5126739Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>
[GitHub] [flink] JingsongLi commented on a change in pull request #17792: [FLINK-24763][fs-connector] LimitableReader should swallow exception when reached limit
JingsongLi commented on a change in pull request #17792:
URL: https://github.com/apache/flink/pull/17792#discussion_r751860456
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/LimitableBulkFormat.java
##
@@ -105,8 +111,16 @@ private boolean reachLimit() {
return null;
}
-RecordIterator batch = reader.readBatch();
-return batch == null ? null : new LimitableIterator(batch);
+try {
+RecordIterator batch = reader.readBatch();
+return batch == null ? null : new LimitableIterator(batch);
+} catch (Exception e) {
Review comment:
I think we can ignore other exceptions too. After all, it has read
enough records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24944) Flink job on k8s mode supports pod scheduling according to priority
[ https://issues.apache.org/jira/browse/FLINK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445601#comment-17445601 ] Yangze Guo commented on FLINK-24944: cc [~wangyang0918] Can you evaluate this requirement? > Flink job on k8s mode supports pod scheduling according to priority > --- > > Key: FLINK-24944 > URL: https://issues.apache.org/jira/browse/FLINK-24944 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: john >Priority: Major > > Hi, is it possible to support the specified pod priority when supporting the > pod scheduling of the job? I think it is necessary to support the priority > scheduling of the job in a cluster with multiple users. In Kubernetes 1.8 and > later versions, the priority of Pod is specified by PriorityClass. At > present, if I want to specify the priority when submitting a job, I must > provide a separate pod-templete.yaml for each job, which is often not > flexible enough. Can we provide a separate parameter similar to > yarn.application.priority to support the priority of the job when we submit > the job to k8s,，such as kubernetes .application.priority-class。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-15825) Add renameDatabase() to Catalog
[
https://issues.apache.org/jira/browse/FLINK-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445597#comment-17445597
]
Jingsong Lee commented on FLINK-15825:
--
[~shenzhu0127] Thanks!
> Add renameDatabase() to Catalog
> ---
>
> Key: FLINK-15825
> URL: https://issues.apache.org/jira/browse/FLINK-15825
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Assignee: Shen Zhu
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> The {{Catalog}} interface lacks a method to rename a database.
> It is possible to change all properties (via {{alterDatabase()}}) but it is
> not possible to rename a database.
> A {{renameTable()}} method is exists.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-15825) Add renameDatabase() to Catalog
[
https://issues.apache.org/jira/browse/FLINK-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee reassigned FLINK-15825:
Assignee: Shen Zhu
> Add renameDatabase() to Catalog
> ---
>
> Key: FLINK-15825
> URL: https://issues.apache.org/jira/browse/FLINK-15825
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Fabian Hueske
>Assignee: Shen Zhu
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> The {{Catalog}} interface lacks a method to rename a database.
> It is possible to change all properties (via {{alterDatabase()}}) but it is
> not possible to rename a database.
> A {{renameTable()}} method is exists.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] SteNicholas commented on pull request #17814: [FLINK-24899][runtime] Enable data compression for blocking shuffle by default
SteNicholas commented on pull request #17814: URL: https://github.com/apache/flink/pull/17814#issuecomment-972457321 @wsry @zhuzhurk , could you please take a look at this pull request? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24942) Could not find any factory for identifier 'hive' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
[
https://issues.apache.org/jira/browse/FLINK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445587#comment-17445587
]
Hang Ruan commented on FLINK-24942:
---
How do you create table `fs_table`? We should make use of `HiveCatalog` to
write to a hive table.
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/hive/hive_read_write/#writing
> Could not find any factory for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
>
>
> Key: FLINK-24942
> URL: https://issues.apache.org/jira/browse/FLINK-24942
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
> Environment: Flink-1.14.0
>Reporter: JasonLee
>Priority: Major
> Fix For: 1.15.0
>
>
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Could not find any factory
> for identifier 'hive' that implements
> 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath.
> Available factory identifiers are:
> blackhole
> datagen
> filesystem
> kafka
> print
> upsert-kafka
>
> The above exception is thrown when I execute the following SQL, even though I
> have added flink-sql-connector-hive-2.3.6_2.11-1.14.0.jar in flink/lib
> {code:java}
> // code placeholder
> insert into fs_table
> select xxx,
> xxx,
> xxx,
> xxx,
> xxx,
> DATE_FORMAT(ts_ltz, '-MM-dd'), DATE_FORMAT(ts_ltz, 'HH')
> from kafka_table; {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24944) Flink job on k8s mode supports pod scheduling according to priority
[ https://issues.apache.org/jira/browse/FLINK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445585#comment-17445585 ] john commented on FLINK-24944: -- [~guoyangze] if I want to specify the priority when submitting a job, I must provide a separate pod-templete.yaml for each job, which is often not flexible enough. > Flink job on k8s mode supports pod scheduling according to priority > --- > > Key: FLINK-24944 > URL: https://issues.apache.org/jira/browse/FLINK-24944 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: john >Priority: Major > > Hi, is it possible to support the specified pod priority when supporting the > pod scheduling of the job? I think it is necessary to support the priority > scheduling of the job in a cluster with multiple users. In Kubernetes 1.8 and > later versions, the priority of Pod is specified by PriorityClass. At > present, if I want to specify the priority when submitting a job, I must > provide a separate pod-templete.yaml for each job, which is often not > flexible enough. Can we provide a separate parameter similar to > yarn.application.priority to support the priority of the job when we submit > the job to k8s,，such as kubernetes .application.priority-class。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24944) Flink job on k8s mode supports pod scheduling according to priority
[ https://issues.apache.org/jira/browse/FLINK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445580#comment-17445580 ] Yangze Guo commented on FLINK-24944: Can pod-template[1] meet your requirement? [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#pod-template > Flink job on k8s mode supports pod scheduling according to priority > --- > > Key: FLINK-24944 > URL: https://issues.apache.org/jira/browse/FLINK-24944 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: john >Priority: Major > > Hi, is it possible to support the specified pod priority when supporting the > pod scheduling of the job? I think it is necessary to support the priority > scheduling of the job in a cluster with multiple users. In Kubernetes 1.8 and > later versions, the priority of Pod is specified by PriorityClass. At > present, if I want to specify the priority when submitting a job, I must > provide a separate pod-templete.yaml for each job, which is often not > flexible enough. Can we provide a separate parameter similar to > yarn.application.priority to support the priority of the job when we submit > the job to k8s,，such as kubernetes .application.priority-class。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24944) Flink job on k8s mode supports pod scheduling according to priority
john created FLINK-24944: Summary: Flink job on k8s mode supports pod scheduling according to priority Key: FLINK-24944 URL: https://issues.apache.org/jira/browse/FLINK-24944 Project: Flink Issue Type: Improvement Components: Deployment / Kubernetes Reporter: john Hi, is it possible to support the specified pod priority when supporting the pod scheduling of the job? I think it is necessary to support the priority scheduling of the job in a cluster with multiple users. In Kubernetes 1.8 and later versions, the priority of Pod is specified by PriorityClass. At present, if I want to specify the priority when submitting a job, I must provide a separate pod-templete.yaml for each job, which is often not flexible enough. Can we provide a separate parameter similar to yarn.application.priority to support the priority of the job when we submit the job to k8s,，such as kubernetes .application.priority-class。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24943) SequenceNumber class is not POJO type
[
https://issues.apache.org/jira/browse/FLINK-24943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445571#comment-17445571
]
Alexander Egorov commented on FLINK-24943:
--
The fix should be quite simple, I don't mind to work on PR
> SequenceNumber class is not POJO type
> -
>
> Key: FLINK-24943
> URL: https://issues.apache.org/jira/browse/FLINK-24943
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.12.5, 1.13.3
>Reporter: Alexander Egorov
>Priority: Minor
>
> SequenceNumber class is currently part of the "Kinesis-Stream-Shard-State",
> but it does not follow requirements of POJO compatible type. Because of that
> we are getting warning like this:
> {{TypeExtractor - class
> software.amazon.kinesis.connectors.flink.model.SequenceNumber does not
> contain a setter for field sequenceNumber}}
> While the warning itself or inability to use optimal sterilizer for such a
> small state is not the problem, this warning prevents us to disable Generic
> Types via {{disableGenericTypes()}}
> So, the problem is similar to
> https://issues.apache.org/jira/browse/FLINK-15904
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24943) SequenceNumber class is not POJO type
[
https://issues.apache.org/jira/browse/FLINK-24943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Egorov updated FLINK-24943:
-
Component/s: Connectors / Kinesis
Affects Version/s: 1.13.3
1.12.5
> SequenceNumber class is not POJO type
> -
>
> Key: FLINK-24943
> URL: https://issues.apache.org/jira/browse/FLINK-24943
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.12.5, 1.13.3
>Reporter: Alexander Egorov
>Priority: Minor
>
> SequenceNumber class is currently part of the "Kinesis-Stream-Shard-State",
> but it does not follow requirements of POJO compatible type. Because of that
> we are getting warning like this:
> {{TypeExtractor - class
> software.amazon.kinesis.connectors.flink.model.SequenceNumber does not
> contain a setter for field sequenceNumber}}
> While the warning itself or inability to use optimal sterilizer for such a
> small state is not the problem, this warning prevents us to disable Generic
> Types via {{disableGenericTypes()}}
> So, the problem is similar to
> https://issues.apache.org/jira/browse/FLINK-15904
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-24943) SequenceNumber class is not POJO type
Alexander Egorov created FLINK-24943:
Summary: SequenceNumber class is not POJO type
Key: FLINK-24943
URL: https://issues.apache.org/jira/browse/FLINK-24943
Project: Flink
Issue Type: Bug
Reporter: Alexander Egorov
SequenceNumber class is currently part of the "Kinesis-Stream-Shard-State", but
it does not follow requirements of POJO compatible type. Because of that we are
getting warning like this:
{{TypeExtractor - class
software.amazon.kinesis.connectors.flink.model.SequenceNumber does not contain
a setter for field sequenceNumber}}
While the warning itself or inability to use optimal sterilizer for such a
small state is not the problem, this warning prevents us to disable Generic
Types via {{disableGenericTypes()}}
So, the problem is similar to https://issues.apache.org/jira/browse/FLINK-15904
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16570) Difficulties to select correct metric with long name in dropdown of Flink UI task menu
[ https://issues.apache.org/jira/browse/FLINK-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16570: --- Labels: auto-deprioritized-major auto-deprioritized-minor (was: auto-deprioritized-major stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Difficulties to select correct metric with long name in dropdown of Flink UI > task menu > -- > > Key: FLINK-16570 > URL: https://issues.apache.org/jira/browse/FLINK-16570 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: Fabian Paul >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > Attachments: metrics_dropdown.png > > > As seen in the attached image it is currently difficult to select the correct > metrics when the metric name exceeds the length of the dropdown because the > full name cannot be seen. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-11862) many differents query on same stream cause second condition of where of second query error
[
https://issues.apache.org/jira/browse/FLINK-11862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11862:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> many differents query on same stream cause second condition of where of
> second query error
> --
>
> Key: FLINK-11862
> URL: https://issues.apache.org/jira/browse/FLINK-11862
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.7.2
> Environment: flink 1.7版本 java 1.8
>Reporter: zhengbm
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> List fields = Lists.newArrayList("rawMessage","timestamp");
> Schema schema = new Schema();
> for (int i = 0; i < fields.size(); i++)
> { schema.field(fields.get(i), Types.STRING()).from(fields.get(i)); }
> tableEnvironment.connect(new Kafka()
> .version("0.8")
> .properties(properties)
> .topic("raw_playtime_h5_source")
> .startFromLatest()
> )
> .withFormat(new Json().failOnMissingField(false).deriveSchema())
> .withSchema(schema)
> .inAppendMode()
> .registerTableSource("t1");
> Table table2 = tableEnvironment
> .sqlQuery("select maps,`timestamp`,CARDINALITY(maps) AS maps_length ,1 as
> flash from t1 ,LATERAL TABLE(split(rawMessage,'
> t')) as T(maps) ");
> tableEnvironment.registerTable("t2", table2);
> Table table = tableEnvironment.sqlQuery("select `timestamp`,maps_length from
> t2 where maps_length>0");
> TypeInformation typeInformation = table.getSchema().toRowType();
> String[] columns = table.getSchema().getFieldNames();
> DataStream dataStream = tableEnvironment
> .toAppendStream(table, typeInformation)
> .map(new PhysicTransformMap(columns, 0));
> dataStream.print();
> try
> { env.execute(); }
> catch (Exception e)
> { e.printStackTrace(); }
> {color:#d04437}noted：message of kafka : \{"timestamp" :
> "","rawMessage":"xxx\txxx\t\t"}（if I delete where condition of second
> query will ok,for "select `timestamp`,maps_length from t2 " or "select
> `timestamp`,maps_length from t2 where timestamp>0" is also ok）{color}
> {color:#d04437}next is trace of exception{color}
> Exception in thread "main" org.apache.flink.table.codegen.CodeGenException:
> Invalid input access.
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$15.apply(CodeGenerator.scala:587)
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$15.apply(CodeGenerator.scala:587)
> at scala.Option.getOrElse(Option.scala:120)
> at
> org.apache.flink.table.codegen.CodeGenerator.visitInputRef(CodeGenerator.scala:587)
> at
> org.apache.flink.table.codegen.CodeGenerator.visitInputRef(CodeGenerator.scala:66)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:754)
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:744)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:744)
> at
> org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:66)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:107)
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:754)
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:744)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
[jira] [Updated] (FLINK-11457) PrometheusPushGatewayReporter does not cleanup its metrics
[
https://issues.apache.org/jira/browse/FLINK-11457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11457:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> PrometheusPushGatewayReporter does not cleanup its metrics
> --
>
> Key: FLINK-11457
> URL: https://issues.apache.org/jira/browse/FLINK-11457
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
>Reporter: Oscar Westra van Holthe - Kind
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> When cancelling a job running on a yarn based cluster and then shutting down
> the cluster, metrics on the push gateway are not deleted.
> My yarn-conf.yaml settings:
> {code:yaml}
> metrics.reporters: promgateway
> metrics.reporter.promgateway.class:
> org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
> metrics.reporter.promgateway.host: pushgateway.gcpstg.bolcom.net
> metrics.reporter.promgateway.port: 9091
> metrics.reporter.promgateway.jobName: PSMF
> metrics.reporter.promgateway.randomJobNameSuffix: true
> metrics.reporter.promgateway.deleteOnShutdown: true
> metrics.reporter.promgateway.interval: 30 SECONDS
> {code}
> What I expect to happen:
> * when running, the metrics are pushed to the push gateway to a separate
> label per node (jobmanager/taskmanager)
> * when shutting down, the metrics are deleted from the push gateway
> This last bit does not happen.
> How the job is run:
> {code}flink run -m yarn-cluster -yn 5 -ys 2 -yst
> "$INSTALL_DIRECTORY/app/psmf.jar"{code}
> How the job is stopped:
> {code}
> YARN_APP_ID=$(yarn application -list | grep "PSMF" | awk '{print $1}')
> FLINK_JOB_ID=$(flink list -r -yid ${YARN_APP_ID} | grep "PSMF" | awk '{print
> $4}')
> flink cancel -s "${SAVEPOINT_DIR%/}/" -yid "${YARN_APP_ID}" "${FLINK_JOB_ID}"
> echo "stop" | yarn-session.sh -id ${YARN_APP_ID}
> {code}
> Is there anything I'm sdoing wrong? Anything I can help to fix?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17811: [FLINK-24754] Implement CHAR/VARCHAR length validation for sinks
flinkbot edited a comment on pull request #17811: URL: https://github.com/apache/flink/pull/17811#issuecomment-970441907 ## CI report: * bb14b0ec0bb661fb900d594dba04ff275a1df664 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26615) * 2b885bb3d72eb7489b8aa3211d63a7c1b7ca591d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26682) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
