Re: flink权限问题

2019-10-17 文章 Wesley Peng 
hi

hdfs有严格的权限管理系统，这个问题似乎绕不过去吧，flink该有的权限就应该给它开了。

On Thu, Oct 17, 2019 at 7:58 PM jinhai wang  wrote:

> 背景：
>
> 实时flink任务从kafka同步数据到hdfs。
>
>
>
> 问题：
>
> 比如flink的job使用A用户提交，但写hdfs的文件用户只能是B，那这个除了给hdfs添加A的操作权限外，flink
> 这块儿可以优雅解决这类问题吗？checkpoint的写权限应该是A用户吧
>

Re: 文件重命名

2019-10-08 文章 Wesley Peng 

May you want to rename them in HDFS with FileSystem.rename method?


on 2019/10/8 17:39, yanggang_it_job wrote:

写入hdfs的文件都是
part-{parallel-task}-{count}
这种格式


如何重命名啊？

Re: 订阅邮件

2019-09-27 文章 Wesley Peng 
Hello

You should not pose across multiple groups.
One thread to one group is more graceful.


杨利君 于2019年9月26日 周四下午3:42写道：

> 订阅flink社区邮件

Re: Split a stream into any number of streams

2019-09-16 文章 Wesley Peng 

Hi

on 2019/9/17 10:28, 王佩 wrote:

*  //  How should I do it?*
splitStream.select("productID1").print();


If I understand for that correctly, you want somewhat the dynamic number 
of Sinks?


regards

Re: Split a stream into any number of streams

2019-09-16 文章 Wesley Peng 




on 2019/9/17 9:55, 王佩 wrote:

I want to split a stream into any number of streams according to a field,
and then process the split stream one by one.


I think that should be easy done. refer to:
https://stackoverflow.com/questions/53588554/apache-flink-using-filter-or-split-to-split-a-stream

regards.

Re: Flink大state读取磁盘，磁盘IO打满，任务相互影响的问题探讨

2019-09-10 文章 Wesley Peng 




on 2019/9/10 13:47, 蒋涛涛 wrote:

尝试手段：

1. 手动迁移IO比较高的任务到其他机器，但是yarn任务提交比较随机，只能偶尔为之

2. 目前没有SSD，只能用普通STATA盘，目前加了两块盘提示磁盘IO能力，但是单盘对单任务的磁盘IO瓶颈还在

还有哪些策略可以解决或者缓解么？


It seems the tricks to improve RocksDB's throughput might be helpfu.

With writes and reads accessing mostly the recent data, our goal is to 
let them stay in memory as much as possible without using up all the 
memory on the server. The following parameters are worth tuning:


Block cache size: When uncompressed blocks are read from SSTables, they 
are cached in memory. The amount of data that can be stored before 
eviction policies apply is determined by the block cache size. The 
bigger the better.


Write buffer size: How big can Memtable get before it is frozen. 
Generally, the bigger the better. The tradeoff is that big write buffer 
takes more memory and longer to flush to disk and to recover.


Write buffer number: How many Memtables to keep before flushing to 
SSTable. Generally, the bigger the better. Similarly, the tradeoff is 
that too many write buffers take up more memory and longer to flush to disk.


Minimum write buffers to merge: If most recently written keys are 
frequently changed, it is better to only flush the latest version to 
SSTable. This parameter controls how many Memtables it will try to merge 
before flushing to SSTable. It should be less than the write buffer 
number. A suggested value is 2. If the number is too big, it takes 
longer to merge buffers and there is less chance of duplicate keys in 
that many buffers.


The list above is far from being exhaustive, but tuning them correctly 
can have a big impact on performance. Please refer to RocksDB’s Tuning 
Guide for more details on these parameters. Figuring out the optimal 
combination of values for all of them is an art in itself.


please ref: https://klaviyo.tech/flinkperf-c7bd28acc67

regards.

Re: FLINK WEEKLY 2019/36

2019-09-08 文章 Wesley Peng 




on 2019/9/9 11:23, Zili Chen wrote:

FLINK WEEKLY 2019/36

很高兴和大家分享上周 FLINK 社区的发展。在过去的一周里，更多 FLINK 1.10 的特性被提出和讨论，包括新的 FLIP，来自 Apache
Pulsar 社区的 Connector 贡献等等。专门讨论 FLINK 1.10 将要实现什么特性的议程也在进行。
用户问题


A nice work. thank you.

regards.

Re: 希望获得一份编译后的1.6版本以上flink源码

2019-09-06 文章 Wesley Peng 

Helo

guaishushu1...@163.com wrote:

因windows系统搭载虚拟机环境，总是不能成功编译flink源码，所以希望能得到一份编译后的1.6版本以上的flink源码


there is docker image for flink 1.9. since you are using a virtual 
machine, then docker might be used.


regards

Re: Flink Savepoint 超时

2019-09-06 文章 Wesley Peng 




SJMSTER wrote:

Checkpoints一直都是成功的。
今天重新尝试了一下cancle job with savepoint又成功了..
不知道之前为什么试了几次都是超时的..


are there any log items for diagnosis?

regards.

Re: flink1.9.0对DDL的支持

2019-09-04 文章 Wesley Peng 

Hi

on 2019/9/5 11:23, pengcheng...@bonc.com.cn wrote:

请教一下， 1.flink1.9.0的table API/sql是不是还没有支持Create view？


from the official documentation of flink 1.9:

Views can also be created within a CLI session using the CREATE VIEW 
statement:


CREATE VIEW MyNewView AS SELECT MyField2 FROM MyTableSource;

Views created within a CLI session can also be removed again using the 
DROP VIEW statement:


DROP VIEW MyNewView;

Attention The definition of views in the CLI is limited to the mentioned 
syntax above. Defining a schema for views or escaping whitespaces in 
table names will be supported in future versions.


So create view is supported but has the limits.

regards.

Re: FLink WEB 怎么加登录验证？

2019-09-04 文章 Wesley Peng 

Hi

on 2019/9/5 10:46, wanghongquan.sh wrote:

FLink WEB 控制台中，没有找到登录验证的配置，请问这个WEB怎么加登录验证？


Flink does not directly support authenticating access to the web UI, but 
you can always put something like Apache's basic_auth in front of it.

Re: flink集群模式下如何动态添加taskmanager

2019-09-03 文章 Wesley Peng 

Hi

on 2019/9/4 10:27, 如影随形 wrote:

  在flink集群部署时，有1个jobmanager,有2个taskmanager(slave1,slave2)，
现在想添加slave3作为 taskmnanager。如何在不停止集群的情况下，可以像spark一样动态添加吗？


AFAIK the answer is NO for now. However, community tells that this has 
been under consideration from the FLIP-6 Flink Development and Process 
Model.


https://docs.google.com/document/d/1zwBP3LKnJ0LI1R4-9hLXBVjOXAkQS5jKrGYYoPz-xRk/edit#

Re: 回复： flink使用StateBackend问题

2019-09-03 文章 Wesley Peng 




on 2019/9/3 15:38, 守护 wrote:

org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received late 
message for now expired checkpoint attempt 3 from 
24674987621178ed1a363901acc5b128 of job fd5010cbf20501339f1136600f0709c3.
请问这个是什么问题呢？


可以根据这些失败的task的id去查询这些任务落在哪一个taskmanager上，经过排查发现，是同一台机器，通过ui看到该机器流入的数据明显比别的流入量大 
因此是因为数据倾斜导致了这个问题，追根溯源还是下游消费能力不足的问题


also reference:
https://juejin.im/post/5c374fe3e51d451bd1663756

Re: flink使用StateBackend问题

2019-09-03 文章 Wesley Peng 

Hi

on 2019/9/3 12:14, 々守护々 wrote:

现象：我是在代码中配置StateBackend的，代码是这样写的env.setStateBackend(new 
RocksDBStateBackend("hdfs://host51:9000/flink/checkpoints",true))，但是我发现在下面的SQLwindow中始终在checkpoint那就过不去，总是报n/a，

下面我就把代码中的这句给注释掉了，在flink-conf.xml文件中配置
state.checkpoints.dir: hdfs://host51:9000/flink/flink-checkpoints这样就能正常执行，
请问SQLwindow和RocksDBStateBackend有什么冲突吗，上面的原因是什么？



It seems like a filesystem issue, though I have no experience on it too.

regards.

Re: FlinkKafkaProducer 开启Excatly Once之后 初始化事务状态超时的问题

2019-09-01 文章 Wesley Peng 

Hi

on 2019/9/2 11:49, 陈赋赟 wrote:

2019-09-02 10:24:28,599 INFO  org.apache.flink.runtime.taskmanager.Task
 - Interval Join -> Sink: Unnamed (1/4) 
(e8b85b6f144879efbb0b4209f226c69b) switched from RUNNING to FAILED.
org.apache.kafka.common.errors.TimeoutException: Timeout expired while 
initializing transactional state in 6ms.


You may reference this:

https://stackoverflow.com/questions/54295588/kafka-streams-failed-to-rebalance-error

Possible options:

As this answer says, switch off Exactly Once for your streamer. It then 
doesn't use transactions and all seems to work ok. Not helpful if you 
require EOS or some other client code requires transactions.
restart any brokers that are reporting warnings to force them to 
re-resolve the IP address. They would need to be restarted in a way that 
they don't change IP address themselves. Not usually possible in kubernetes.
Defect raised Issue KAFKA-7958 - Transactions are broken with kubernetes 
hosted brokers


Update 2017-02-20 This may have been resolved in Kafka 2.1.1 (Confluent 
5.1.2) released today. See the linked issue.

Re: 关于flink 写于kafka时的transactionId 生成问题

2019-08-29 文章 Wesley Peng 

Hi

on 2019/8/29 17:50, ddwcg wrote:

broker就一个，flink集群的时钟确实和broker的不一样，是不同的机房，能自己指定transactionalId吗，两个机房的调整成一样怕影响其他的应用


AFAIK the transID is generated by systems.

regards.

Re: 关于flink 写于kafka时的transactionId 生成问题

2019-08-29 文章 Wesley Peng 

Hi

on 2019/8/29 17:13, ddwcg wrote:

作业我已经停止了，但是看kafka的日志还是不断的在刷Initialized transactionalId………. ，而且该程序再此启动就会报：
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer 
attempted an operation with an old epoch. Either there is a newer producer with 
the same transactionalId, or the producer's transaction has been expired by the 
broker.
错误，请问有什么办法可以避免这个问题？


Maybe you want to check all the broker and producers have the same 
timezone setup, and all time are synchronous.


regards.