Re: Re:[进度更新] [讨论] Flink 对 Hive 的兼容 和 Catalogs

感谢大家的回复！下一步我会整理好各位的反馈并转达给我们的团队。 同时欢迎加入 Flink-Hive 官方用户钉钉群讨论和反馈问题 [image: image.png] On Wed, Mar 20, 2019 at 8:39 AM ChangTong He wrote: > >- *各位在使用哪个版本的Hive？有计划升级Hive吗？* > > 目前我维护的两套批处理系统分别是CDH5.10.0 > >

Re: Source reinterpretAsKeyedStream

Hi Adrienne, I think you should be able to reinterpretAsKeyedStream by passing in a DataStreamSource based on the ITCase example [1]. Can you share the full code/error logs or the IAE? -- Rong [1]

Re: Infinitely requesting for Yarn container in Flink 1.5

Hi Qi, I think the problem may be related to another similar problem reported in a previous JIRA [1]. I think a PR is also in discussion. Thanks, Rong [1] https://issues.apache.org/jira/browse/FLINK-10868 On Fri, Mar 29, 2019 at 5:09 AM qi luo wrote: > Hello, > > Today we encountered an

Re: Calcite SQL Map to Pojo Map

I think the proper solution should not be Types.GENERIC(Map.class) as you will not be able to do any success processing with the return object. For example, Map['k', 'v'].get('k') will not work. I think there might be some problem like you suggested that they are handled as GenericType instead of

Re: StreamingFileSink seems to be overwriting existing part files

No problem! Cheers, Kostas On Fri, Mar 29, 2019 at 4:38 PM Bruno Aranda wrote: > Hi Kostas, > > Put that way, sounds fair enough. Many thanks for the clarification, > > Cheers, > > Bruno > > On Fri, 29 Mar 2019 at 15:32, Kostas Kloudas wrote: > >> Hi Bruno, >> >> This is the expected

Re: StreamingFileSink seems to be overwriting existing part files

Hi Kostas, Put that way, sounds fair enough. Many thanks for the clarification, Cheers, Bruno On Fri, 29 Mar 2019 at 15:32, Kostas Kloudas wrote: > Hi Bruno, > > This is the expected behaviour as the job starts "fresh", given that you > did not specify any savepoint/checkpoint to start from.

Re: StreamingFileSink seems to be overwriting existing part files

Hi Bruno, This is the expected behaviour as the job starts "fresh", given that you did not specify any savepoint/checkpoint to start from. As for the note that "One would expect that it finds the last part and gets the next free number?", I am not sure how this can be achieved safely and

StreamingFileSink seems to be overwriting existing part files

Hi, One of the main reasons we moved to version 1.7 (and 1.7.2 in particular) was because of the possibility of using a StreamingFileSink with S3. We've configured a StreamingFileSink to use a DateTimeBucketAssigner to bucket by day. It's got a parallelism of 1 and is writing to S3 from an EMR

Re: How to run a job with job cluster mode on top of mesos?

Thanks a lot Jacky. Cheers, Till On Fri, Mar 29, 2019 at 1:50 PM Jacky Yin 殷传旺 wrote: > Hello Till, > > > > Thanks very much for the clear steps. And I noticed that you have already > created a jira ticket for this issue and I would like to contribute it. (I > have already assigned it to me).

答复: blink开源版本维表关联时开启缓存方式

感谢大佬解答，我尝试了一下已经基本实现了缓存。期待能早日用上flink原生的维表缓存功能 发送自 Windows 10 版邮件应用 发件人: Kurt Young 发送时间: Friday, March 29, 2019 5:33:57 PM 收件人: user-zh@flink.apache.org 主题: Re: blink开源版本维表关联时开启缓存方式

Re: How can I visualize the metrics Counter and Meter at VisualVM?

Did you explicitly connector to the specific host/port, or just to a local process? (If you specify a port, you must connect to the specific port) On 29/03/2019 15:25, Felipe Gutierrez wrote: Hi, I have configured JMXReporter on Flink. I have a RichMapFunction class emitting a counter and a

How can I visualize the metrics Counter and Meter at VisualVM?

Hi, I have configured JMXReporter on Flink. I have a RichMapFunction class emitting a counter and a meter . When I open VisualVM and the MBeans tab I

Source reinterpretAsKeyedStream

Dear community, I have a use-case where sources are keyed. For example, there is a source function with parallelism 10, and each instance has its own key. I used reinterpretAsKeyedStream to convert source DataStream to KeyedStream, however, I get an IllegalArgument exception. Is

Re: Do we have an example of setting up Queryable state ( proxies, client etc ) on k8s ?

Thanks Konstantin, That makes sense. To give you some context, the reason we are gravitating towards Queryable State is the architectural preference of Prometheus to scrape ( pull rather then push model ) and our intent to expose aggregations. That said your idea makes

Re: Support for custom triggers in Table / SQL

Hi Piyush, Custom triggers (or early firing) is currently not supported by SQL or the Table API. It is also not on the roadmap [1]. Currently, most efforts on the relational API are focused on restructuring the code and working towards the integration of the Blink contribution [2]. AFAIK, there

Re: 远程提交代码到Flink集群

Flink的ClassLoader就是普通的URLClassLoader，只是加载顺序是ChildFirst。 不建议你用直接更新classes文件去解决版本更新问题，因为你可能用了很大的力气去解决一个伪需求。 可以用脚本通过Flink提供的rest 接口来实现jar包上传等动作，甚至可以将CI和部署结合起来。 我曾今做过类似的事情，你可以参考下： https://blog.csdn.net/xianzhen376/article/details/88529527 On 2019/03/28 13:06:41, "文报" <1...@qq.com> wrote: > 各位好！> >

??????Flink Job ????

job created??running??cancelling??canceled??failed??RESTARTING job -- -- ??: "cheng"; : 2019??3??28??(??) 4:38 ??: "user-zh"; : Flink Job

??????????Blink ????kafka??????????

-- -- ??: "qq??"; : 2019??3??28??(??) 5:02 ??: "user-zh"; : Blink kafka?? ??Blink kafka 30

?????? Flink Job ????

?? flink ui job ?? job ?? -- -- ??: "cheng"; : 2019??3??28??(??) 5:08 ??: "user-zh"; : Re: Flink Job standalone job state

Re: How to run a job with job cluster mode on top of mesos?

Hello Till, Thanks very much for the clear steps. And I noticed that you have already created a jira ticket for this issue and I would like to contribute it. (I have already assigned it to me).  Jacky Yin 发件人: Till Rohrmann 日期: 2019年3月26日 星期二 下午6:31 收件人: Jacky Yin 殷传旺 抄送:

【问题】inputqueueLength/inputqueueLength一直和并行度一样？怎么降低这个值

Infinitely requesting for Yarn container in Flink 1.5

Hello, Today we encountered an issue where our Flink job request for Yarn container infinitely. In the JM log as below, there were errors when starting TMs (caused by underlying HDFS errors). So the allocated container failed and the job kept requesting for new containers. The failed

Re: 结邮 Re: Re: 请教一下Blink资源分配问题

Blink是基于Flink 1.5.1做的二次开发，可能存在部分配置和最新版Flink不一致的情况。Sorry 让你踩坑了。 Best, Kurt On Fri, Mar 29, 2019 at 5:52 PM 邓成刚【qq】 wrote: > 终于发现是什么问题了，是由于Blink的 配置与FLINK不同导致： > Flink 里没有这个配置： taskmanager.cpu.core 默认是 1 > > 另外：blink 里 taskmanager.heap.mb 与 flink 的 taskmanager.heap.size 不同导致 >

结邮 Re: Re: 请教一下Blink资源分配问题

终于发现是什么问题了，是由于Blink的 配置与FLINK不同导致： Flink 里没有这个配置：  taskmanager.cpu.core  默认是   1    另外：blink 里 taskmanager.heap.mb   与 flink 的 taskmanager.heap.size  不同导致    taskmanager.heap 配置过小，默认1G 之前错误的配置： # The heap size for the JobManager JVM jobmanager.heap.size: 20480m # The heap size for the

Commercial support for Apache Flink

Hello Team, Can you please let us know who all are the companies provides the commercial support for Apache Flink. One we know i.e. Data Artisans. Please let me know if any other company provides the same? [Ericsson] DEEPTI SHARMA Specialist ITIL 2011 Foundation

Re: Do we have an example of setting up Queryable state ( proxies, client etc ) on k8s ?

Hi Vishal, my approach would be a single Kubernetes service, which is backed by all Taskmanagers of the job. The Taskmanagers will proxy the request for a specific key to the correct Taskmanager. Yes, the Taskmanagers will cache the location of the key groups. In addition to this Kubernetes

Re:远程提交代码到Flink集群

env.createRemoteEnviroment(); 在 2019-03-28 21:06:41，"文报" <1010467...@qq.com> 写道： >各位好！ > > 请教一下各位，在本地开发完代码后，怎么样可以将编写好的代码直接提交到Flink集群上运行?(想做Flink任务的自动化，避免每次开发完成后都需要打jar包提交到web页面上)

Re: Re: 请教一下Blink资源分配问题

taskmanager.heap.mb taskmanager.cpu.core taskmanager.direct.memory.mb 这几个参数你都配置的多少贴下？ Best, Guowei 邓成刚【qq】 于2019年3月29日周五 下午5:18写道： > 用的是table ， LOG里资源申请情况： > > ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=2,

Re: blink开源版本维表关联时开启缓存方式

当时没有想清楚如何把Cache当成一个public的接口向外提供，它更像是一些实现上的特定优化。 后续我们在flink master上实现维表join的时候，会把这个问题考虑进去。 Best, Kurt On Fri, Mar 29, 2019 at 5:09 PM moxian wrote: > 这么好的一个优化，为啥被拿掉了呢？ > > Kurt Young 于2019年3月29日周五 上午9:39写道： > > > Hi, > > > > Blink开源的时候把Cache的实现暂时拿掉了，你可以根据自己的需要自己实现一个cache。 > > > > Best, > >

Re: Re: 请教一下Blink资源分配问题

用的是table ， LOG里资源申请情况： ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0, FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request = 5ac9229acae1e6ef90563a5a0bf3fe21). ResourceProfile{cpuCores=0.6,

Re: Re: 请教一下Blink资源分配问题

用的什么api，DataStream还是Table？ 如果是DataStream的话，申请什么样的资源？ 详细列下。 Best, Guowei 邓成刚【qq】 于2019年3月29日周五 下午5:09写道： > 是的。 > > 发件人： moxian > 发送时间： 2019-03-29 17:06 > 收件人： user-zh > 主题： Re: 请教一下Blink资源分配问题 > standalone 模式？ > > 邓成刚【qq】 于2019年3月29日周五 上午9:59写道： > > > 请教一下Blink资源分配问题： > > blink 任务并行度设置 20

Re: blink开源版本维表关联时开启缓存方式

这么好的一个优化，为啥被拿掉了呢？ Kurt Young 于2019年3月29日周五 上午9:39写道： > Hi, > > Blink开源的时候把Cache的实现暂时拿掉了，你可以根据自己的需要自己实现一个cache。 > > Best, > Kurt > > > On Wed, Mar 27, 2019 at 4:44 PM 苏 欣 wrote: > > > 我在ppt里面看到这些内容，但是在开源的blink里面没有找到相关的配置，请问各位老师应该如何开启缓存策略？ > > > > > > > > 发送自 Windows 10 版邮件

Re: Re: 请教一下Blink资源分配问题

是的。 发件人： moxian 发送时间： 2019-03-29 17:06 收件人： user-zh 主题： Re: 请教一下Blink资源分配问题 standalone 模式？   邓成刚【qq】 于2019年3月29日周五 上午9:59写道：   > 请教一下Blink资源分配问题： > blink 任务并行度设置 20  提示0个满足：Batch request 40 slots, but only 0 are fulfilled. > 调整到 3 并行度 提示：Batch request 6 slots, but only 4 are fulfilled.， >

Re: 请教一下Blink资源分配问题

standalone 模式？ 邓成刚【qq】 于2019年3月29日周五 上午9:59写道： > 请教一下Blink资源分配问题： > blink 任务并行度设置 20 提示0个满足：Batch request 40 slots, but only 0 are fulfilled. > 调整到 3 并行度 提示：Batch request 6 slots, but only 4 are fulfilled.， > 但是我的TASK SLOTS有配 48，没有其它任务， > 按理説没有资源问题啊,集群配置情况： > > 其它的都是默认配置： > >

答复: RocksDB中指定nameNode 的高可用

可以了，感谢指教。 发件人: Biao Liu 发送时间: 2019年3月27日 19:13 收件人: user-zh@flink.apache.org 主题: Re: RocksDB中指定nameNode 的高可用 Hi, HDFS 本身可以解决该问题，可以搜一下 “HDFS HA nameservice”，可以避免写死 name node 地址 Yun Tang 于2019年3月26日周二 下午5:29写道： > Hi > > Flink高可用相关配置的存储目录，当存储路径配置成HDFS时，相关namenode高可用性由HDFS支持，对上层完全透明。 > > 祝好 >

?????? ??????????????Flink????

-- -- ??: "Lifei Chen"; : 2019??3??29??(??) 11:10 ??: "user-zh"; : Re: ??Flink go cli, jarflink manager https://github.com/ing-bank/flink-deployer