回复: apache-flink

[wukon...@foxmail.com]

Hi 
  邮件组里没有看到你发的附件 和截图  可以尝试引用外部文档链接方式 来描述问题
 
发件人： 杜畅
发送时间： 2021-09-06 10:43
收件人： user-zh
主题： apache-flink
 flink 1.13.2 流作业消费kafka消息 实时存储到hive中，作业平台发现job第三步没有消息sent 
从网上搜索均无此解决方案，代码也是从官网仿照的。
望帮看下是什么原因

截图和代码请见附件


 

Re: [ANNOUNCE] Apache Flink Stateful Functions 3.1.0 released

[Arvid Heise]

Congratulations! New features look awesome.

On Wed, Sep 1, 2021 at 9:10 AM Till Rohrmann  wrote:

> Great news! Thanks a lot for all your work on the new release :-)
>
> Cheers,
> Till
>
> On Wed, Sep 1, 2021 at 9:07 AM Johannes Moser  wrote:
>
>> Congratulations, great job. 
>>
>> On 31.08.2021, at 17:09, Igal Shilman  wrote:
>>
>> The Apache Flink community is very happy to announce the release of Apache
>> Flink Stateful Functions (StateFun) 3.1.0.
>>
>> StateFun is a cross-platform stack for building Stateful Serverless
>> applications, making it radically simpler to develop scalable, consistent,
>> and elastic distributed applications.
>>
>> Please check out the release blog post for an overview of the release:
>> https://flink.apache.org/news/2021/08/31/release-statefun-3.1.0.html
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Maven artifacts for StateFun can be found at:
>> https://search.maven.org/search?q=g:org.apache.flink%20statefun
>>
>> Python SDK for StateFun published to the PyPI index can be found at:
>> https://pypi.org/project/apache-flink-statefun/
>>
>> Official Docker images for StateFun are published to Docker Hub:
>> https://hub.docker.com/r/apache/flink-statefun
>>
>> The full release notes are available in Jira:
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12350038=12315522
>>
>> We would like to thank all contributors of the Apache Flink community who
>> made this release possible!
>>
>> Thanks,
>> Igal
>>
>>
>>

Re: [ANNOUNCE] Flink mailing lists archive service has migrated to Apache Archive service

[JING ZHANG]

Thanks Leonard for driving this.
The information is helpful.

Best,
JING ZHANG

Jark Wu  于2021年9月6日周一 下午4:59写道：

> Thanks Leonard,
>
> I have seen many users complaining that the Flink mailing list doesn't
> work (they were using Nabble).
> I think this information would be very helpful.
>
> Best,
> Jark
>
> On Mon, 6 Sept 2021 at 16:39, Leonard Xu  wrote:
>
>> Hi, all
>>
>> The mailing list archive service Nabble Archive was broken at the end of
>> June, the Flink community has migrated the mailing lists archives[1] to
>> Apache Archive service by commit[2], you can refer [3] to know more mailing
>> lists archives of Flink.
>>
>> Apache Archive service is maintained by ASF thus the stability is
>> guaranteed, it’s a web-based mail archive service which allows you to
>> browse, search, interact, subscribe, unsubscribe, etc. with mailing lists.
>>
>> Apache Archive service shows mails of the last month by default, you can
>> specify the date range to browse, search the history mails.
>>
>>
>> Hope it would be helpful.
>>
>> Best,
>> Leonard
>>
>> [1] The Flink mailing lists in Apache archive service
>> dev mailing list archives:
>> https://lists.apache.org/list.html?d...@flink.apache.org
>> user mailing list archives :
>> https://lists.apache.org/list.html?u...@flink.apache.org
>> user-zh mailing list archives :
>> https://lists.apache.org/list.html?user-zh@flink.apache.org
>> [2]
>> https://github.com/apache/flink-web/commit/9194dda862da00d93f627fd315056471657655d1
>> [3] https://flink.apache.org/community.html#mailing-lists
>>
>

Re: flink-connector-postgres-cdc：no changes will be captured 无数据捕获到

[Leonard Xu]

没看到你的附件呢，你也可以在Flink CDC 项目里建个issue, 图贴 issue里，
Github 贴图比邮件方便点

> 在 2021年9月6日，19:08，Fisher Xiang  写道：
> 
> Thank u Leonard.
> 我把运行日志放在附件了，麻烦看下
> 使用的用户是：role 'repl' [superuser: true, replication: true, inherit: true, create 
> role: false, create db: false, can log in: true]
> 
> 
> BR
> Fisher
> 
> 
> On Mon, Sep 6, 2021 at 5:55 PM Leonard Xu  > wrote:
> Hello, Fisher
> 
> 图挂了，可以用图床工具贴下吗？
> 我可以帮忙看看
> 
> 祝好，
> Leonard
> 
> > 在 2021年9月6日，17:48，Fisher Xiang  > > 写道：
> > 
> > hi,
> > 
> > 在使用  flink-connector-postgres-cdc时（版本从1.1.1 ~ 1.4.0都试过）， 出现一个警告：
> > WARN io.debezium.relational.RelationalDatabaseSchema - After applying the 
> > include/exclude list filters, no changes will be captured. Please check 
> > your configuration!
> > 启动配置是,Starting PostgresConnectorTask with configuration ：
> > 
> > 
> > 
> > 然后，往这些表里面（  table.whitelist = stud,data_input,data_output 
> > ）insert和delete记录时，没有捕获到变更数据，只有以下输出，求看是什么问题：
> > 
> > 
> > 
> > 使用的用户是：role 'repl' [superuser: true, replication: true, inherit: true, 
> > create role: false, create db: false, can log in: true]
> > 
> > BR
> > Fisher
> 

Re: flink-connector-postgres-cdc：no changes will be captured 无数据捕获到

[Fisher Xiang]

Thank u Leonard.
我把运行日志放在附件了，麻烦看下
使用的用户是：role 'repl' [superuser: true, replication: true, inherit: true,
create role: false, create db: false, can log in: true]


BR
Fisher


On Mon, Sep 6, 2021 at 5:55 PM Leonard Xu  wrote:

> Hello, Fisher
>
> 图挂了，可以用图床工具贴下吗？
> 我可以帮忙看看
>
> 祝好，
> Leonard
>
> > 在 2021年9月6日，17:48，Fisher Xiang  写道：
> >
> > hi,
> >
> > 在使用  flink-connector-postgres-cdc时（版本从1.1.1 ~ 1.4.0都试过）， 出现一个警告：
> > WARN io.debezium.relational.RelationalDatabaseSchema - After applying
> the include/exclude list filters, no changes will be captured. Please check
> your configuration!
> > 启动配置是,Starting PostgresConnectorTask with configuration ：
> >
> >
> >
> > 然后，往这些表里面（  table.whitelist = stud,data_input,data_output
> ）insert和delete记录时，没有捕获到变更数据，只有以下输出，求看是什么问题：
> >
> >
> >
> > 使用的用户是：role 'repl' [superuser: true, replication: true, inherit: true,
> create role: false, create db: false, can log in: true]
> >
> > BR
> > Fisher
>
>

Re: flink-connector-postgres-cdc：no changes will be captured 无数据捕获到

[Leonard Xu]

Hello, Fisher

图挂了，可以用图床工具贴下吗？
我可以帮忙看看

祝好，
Leonard

> 在 2021年9月6日，17:48，Fisher Xiang  写道：
> 
> hi,
> 
> 在使用  flink-connector-postgres-cdc时（版本从1.1.1 ~ 1.4.0都试过）， 出现一个警告：
> WARN io.debezium.relational.RelationalDatabaseSchema - After applying the 
> include/exclude list filters, no changes will be captured. Please check your 
> configuration!
> 启动配置是,Starting PostgresConnectorTask with configuration ：
> 
> 
> 
> 然后，往这些表里面（  table.whitelist = stud,data_input,data_output 
> ）insert和delete记录时，没有捕获到变更数据，只有以下输出，求看是什么问题：
> 
> 
> 
> 使用的用户是：role 'repl' [superuser: true, replication: true, inherit: true, create 
> role: false, create db: false, can log in: true]
> 
> BR
> Fisher

flink-connector-postgres-cdc：no changes will be captured 无数据捕获到

[Fisher Xiang]

hi,

在使用  flink-connector-postgres-cdc时（版本从1.1.1 ~ 1.4.0都试过）， 出现一个警告：
WARN io.debezium.relational.RelationalDatabaseSchema - After applying the
include/exclude list filters, no changes will be captured. Please check
your configuration!
启动配置是,Starting PostgresConnectorTask with configuration ：
[image: image.png]


然后，往这些表里面（  table.whitelist = stud,data_input,data_output
）insert和delete记录时，没有捕获到变更数据，只有以下输出，求看是什么问题：

[image: image.png]

使用的用户是：role 'repl' [superuser: true, replication: true, inherit: true,
create role: false, create db: false, can log in: true]

BR
Fisher

?????? flink 1.13.1 ????hive????????????

[Asahi Lee]

2021-09-06 11:20:32,787 WARN 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-192]: 
Failed to execute [select "COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", 
"LONG_HIGH_VALUE", "DOUBLE_LOW_VALUE", "DOUBLE_HIGH_VALUE", 
"BIG_DECIMAL_LOW_VALUE", "BIG_DECIMAL_HIGH_VALUE", "NUM_NULLS", 
"NUM_DISTINCTS", "AVG_COL_LEN", "MAX_COL_LEN", "NUM_TRUES", "NUM_FALSES", 
"LAST_ANALYZED" from "TAB_COL_STATS" where "DB_NAME" = ? and 
"TABLE_NAME" = ? and "COLUMN_NAME" in (?,?)] with parameters [cosldatacenter, 
dw_zyxx_test, a, ??]
javax.jdo.JDODataStoreException: Error executing SQL query "select 
"COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE", 
"DOUBLE_LOW_VALUE", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", 
"BIG_DECIMAL_HIGH_VALUE", "NUM_NULLS", "NUM_DISTINCTS", "AVG_COL_LEN", 
"MAX_COL_LEN", "NUM_TRUES", "NUM_FALSES", "LAST_ANALYZED" from 
"TAB_COL_STATS" where "DB_NAME" = ? and "TABLE_NAME" = ? and 
"COLUMN_NAME" in (?,?)".
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
at 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1611)
at 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$1300(MetaStoreDirectSql.java:83)
at 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql$13.run(MetaStoreDirectSql.java:1173)
at 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.runBatched(MetaStoreDirectSql.java:1656)
at 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:1183)
at 
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6590)
at 
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6587)
at 
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2616)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6586)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6580)
at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
at com.sun.proxy.$Proxy7.getTableColumnStatistics(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:4644)
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy9.get_table_statistics_req(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:11028)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:11012)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
NestedThrowablesStackTrace:
java.sql.SQLException: Illegal mix of collations (latin1_bin,IMPLICIT), 
(utf8mb4_general_ci,COERCIBLE), (utf8mb4_general_ci,COERCIBLE) for operation ' 
IN '
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3933)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3869)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at 

Re: [ANNOUNCE] Flink mailing lists archive service has migrated to Apache Archive service

[Jark Wu]

Thanks Leonard,

I have seen many users complaining that the Flink mailing list doesn't
work (they were using Nabble).
I think this information would be very helpful.

Best,
Jark

On Mon, 6 Sept 2021 at 16:39, Leonard Xu  wrote:

> Hi, all
>
> The mailing list archive service Nabble Archive was broken at the end of
> June, the Flink community has migrated the mailing lists archives[1] to
> Apache Archive service by commit[2], you can refer [3] to know more mailing
> lists archives of Flink.
>
> Apache Archive service is maintained by ASF thus the stability is
> guaranteed, it’s a web-based mail archive service which allows you to
> browse, search, interact, subscribe, unsubscribe, etc. with mailing lists.
>
> Apache Archive service shows mails of the last month by default, you can
> specify the date range to browse, search the history mails.
>
>
> Hope it would be helpful.
>
> Best,
> Leonard
>
> [1] The Flink mailing lists in Apache archive service
> dev mailing list archives:
> https://lists.apache.org/list.html?d...@flink.apache.org
> user mailing list archives :
> https://lists.apache.org/list.html?u...@flink.apache.org
> user-zh mailing list archives :
> https://lists.apache.org/list.html?user-zh@flink.apache.org
> [2]
> https://github.com/apache/flink-web/commit/9194dda862da00d93f627fd315056471657655d1
> [3] https://flink.apache.org/community.html#mailing-lists
>

[ANNOUNCE] Flink mailing lists archive service has migrated to Apache Archive service

[Leonard Xu]

Hi, all

The mailing list archive service Nabble Archive was broken at the end of June, 
the Flink community has migrated the mailing lists archives[1] to Apache 
Archive service by commit[2], you can refer [3] to know more mailing lists 
archives of Flink.

Apache Archive service is maintained by ASF thus the stability is guaranteed, 
it’s a web-based mail archive service which allows you to browse, search, 
interact, subscribe, unsubscribe, etc. with mailing lists.

Apache Archive service shows mails of the last month by default, you can 
specify the date range to browse, search the history mails.  


Hope it would be helpful.

Best,
Leonard

[1] The Flink mailing lists in Apache archive service
dev mailing list archives: 
https://lists.apache.org/list.html?d...@flink.apache.org 
user mailing list archives : 
https://lists.apache.org/list.html?u...@flink.apache.org  
user-zh mailing list archives : 
https://lists.apache.org/list.html?user-zh@flink.apache.org
[2] 
https://github.com/apache/flink-web/commit/9194dda862da00d93f627fd315056471657655d1
[3] https://flink.apache.org/community.html#mailing-lists

flink sink es??????????

[??????]

??
  flinkkafka??sink 
es??kafka??BulkFlushMaxActions??500050


2021-09-05 10:28:24,737 WARN 
org.apache.flink.runtime.taskmanager.Task 
 [] - Task 'Sink:  (19/21)' did not 
react to cancelling signal for 30 seconds, but is stuck in method:
org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:286)
org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:271)
org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:267)
org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:253)
org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6BulkProcessorIndexer.add(Elasticsearch6BulkProcessorIndexer.java:72)
ideal.mousika.job.operator.sink.ElasticsearchSinkNode$1.process(ElasticsearchSinkNode.java:145)
ideal.mousika.job.operator.sink.ElasticsearchSinkNode$1.process(ElasticsearchSinkNode.java:86)
org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.invoke(ElasticsearchSinkBase.java:310)
org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:161)
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178)
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153)
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:351)
org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$169/63670.runDefaultAction(Unknown
 Source)
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:191)
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:566)
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:536)
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
java.lang.Thread.run(Thread.java:745)