Re: `env.java.opts` not persisting after job canceled or failed and then restarted

This may be caused by a jvm process can only load a so once.So a triky way is to rename it。 发自我的 iPhone > 在 2019年1月25日，上午7:12，Aaron Levin 写道： > > Hi Ufuk, > > Update: I've pinned down the issue. It's multiple classloaders loading > `libhadoop.so`: > > ``` > failed to load native hadoop

Re: Install flink-1.7.2 on Azure with Hadoop 2.7 failed

The exception means that yarn expect the the timestamp of the slf4j-log4j12-1.7.15.jar is "2019-02-11 22:32:52" but yarn find the slf4j-log4j12-1.7.15.jar

Re: Apache Flink - Question about broadcast state pattern usage

Hi 1. I think you could use "Using Managed Operator State"[1] (context.getOperatorStateStore().getBroadcastState()) to use the BroadCastState. But you must use it very carefully and guarantee the semantics of broadcast state yourself. I think "The Broadcast State Pattern"[2] is some best practice

Re: checkpointing when yarn session crashed

Could you give more details? Such as which flink version do you use? which Statebackend do you use? Does there has any successful checkpoint? and so on.. I can't reproduce your problem. (I used BucketingSinkTestProgram(enable external checkpoint) + Flink 1.7.2 and default StateBackend ) Best,

Re: flink on yarn log rolling

Could you share the content of log4j.properties file you modified? And which version of Flink are you using? Best Guowei Shengnan YU 于2019年3月20日周三 下午3:30写道： > Hi all: > I'd like to enable log rolling for flink on yarn. I tried to modify > log4j.properties and logback.xml in flink/conf however

Re: Apache Flink - How to destroy global window and release it's resources

Hi, I think you could return a proper TriggerResult, which defines how to deal with the window elements after computing a window in your trigger implementation. You could find the detail information from the doc[1]. 1.

Re: Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

If your implementation only commits your changing after the complete of a checkpoint I think the latency of e2e is at least the interval of checkpoint. I think the document wants to say that a topology, which only has flatmap/filter/map(no task has more than one input) could achieve the exactly

Re: Timestamp Watermark Assigner bpund question

Hi, 1. From doc[1], A Watermark(t) declares that event time has reached time t in that stream, meaning that there should be no more elements from the stream with a timestamp t’ <= t (i.e. events with timestamps older or equal to the watermark). So I think it might be counterintuitive that

Re: flink 1.7.2 freezes, waiting indefinitely for the buffer availability

Hi , Could you jstak the downstream Task (the Window) and have a look at what the window operator is doing? Best, Guowei Rahul Jain 于2019年4月10日周三 下午1:04写道： > We are also seeing something very similar. Looks like a bug. > > It seems to get stuck in LocalBufferPool forever and the job has to be

Re: [ANNOUNCE] Apache Flink 1.8.0 released

Congratulations! Thanks Aljoscha and all contributors! Best, Guowei Jark Wu 于2019年4月10日周三 下午5:47写道： > Cheers! > > Thanks Aljoscha and all others who make 1.8.0 possible. > > On Wed, 10 Apr 2019 at 17:33, vino yang wrote: > > > Great news! > > > > Thanks Aljoscha for being the release

Re: Flink 1.7.2 UI : Jobs removed from Completed Jobs section

I am not very sure about this problem. But you could try to increase jobstore.expiration-time in config. Best, Guowei Jins George 于2019年4月10日周三 下午1:01写道： > Any input on this UI behavior ? > > > > Thanks, > > Jins > > > > *From: *Timothy Victor > *Date: *Monday, April 8, 2019 at 10:47 AM >

Re: Timestamp Watermark Assigner bpund question

never goes backwards ? > > TIA, > > Vijay > > > > On Tue, Apr 9, 2019 at 10:50 PM Guowei Ma wrote: > >> Hi, >> 1. From doc[1], A Watermark(t) declares that event time has reached time >> t in that stream, meaning that there should be no more elements from t

Re: Timestamp Watermark Assigner bpund question

, Guowei Guowei Ma 于2019年4月15日周一 上午9:44写道： > Hi, Vijay > > >>>Then the Operator progresses to the next Watermark as a starting point > for events after event time reaches currWatermark ? > AFAIK, the operator that generates watermark is called by the frame work. >

Re: Queryable state support in Flink 1.9

Hi, I think you should check TM log first and check if there are some info like: 1430 [main] INFO org.apache.flink.queryablestate.server.KvStateServerImpl - Started Queryable State Server @ /127.0.0.1:9069. 1436 [main] INFO org.apache.flink.queryablestate.client.proxy.KvStateClientProxyImpl -

Re: Queryable state support in Flink 1.9

t I am asking is slightly different: > 1. Does Mini cluster support queryable state? > 2. If the answer is yes, how to set it up? > > Boris Lublinsky > FDP Architect > boris.lublin...@lightbend.com > https://www.lightbend.com/ > > On Apr 15, 2019, at 12:07 AM, Guowei Ma wr

Re: How to verify what maxParallelism is set to?

Hi, StreamExecutionEnvironment is used to set a default maxParallelism for global. If a "operator"'s maxParallelism is -1 the operator will be set the maxParallelism which is set by StreamExecutionEnvironment. >>>Any API or way I can verify? I can't find any easy way to do that. But you could use

Re: Timestamp and key preservation over operators

Hi, Most operators will preserve the input elements timestamp if it has. Window is a special case. The timestamp of elements emitted by window is the maxTimestamp of the Window which is triggered. Different Window will have different implementation.(GlobalWindow/TimeWindow/CustomizedWindow). Keyby

Re: assignTimestampsAndWatermarks not work after KeyedStream.process

Hi, After keyby maybe only some of BoundedOutOfOrdernessTimestampExtractors could receive the elements(trip). If that is the case BoundedOutOfOrdernessTimestampExtractor, which does not receive element would not send the WM. Since that the timeWindowAll operator could not be triggered. You could

Re: RichAsyncFunction Timer Service

Hi AFAIK, TimeService is Flink could guarantee the semastics of Best, Guowei Mikhail Pryakhin 于2019年4月26日周五 下午7:57写道： > Hi David, > Thank you! > > Yes, fair enough, but take for instance a BucketingSink class[1], it is a > RichFunction which employs Timeservice to execute time-based logic,

Re: RichAsyncFunction Timer Service

Hi, AFAIK, TimeService in Flink could guarantee the semantics of "at least once/exactly once" after failure. If you only want to reload config periodically I think you could use Java native Timer yourself. Best, Guowei Guowei Ma 于2019年4月28日周日 上午9:25写道： > Hi > AFAIK, Time

Re: assignTimestampsAndWatermarks not work after KeyedStream.process

Time.days(1)); > ``` > > It makes no sense to me. Please help me understand why it doesn't work. > Thanks! > > On 2019/04/19 04:14:31, Guowei Ma wrote: > > Hi, > > After keyby maybe only some of BoundedOutOfOrdernessTimestampExtractors > > could receive

Re: PatternFlatSelectAdapter - Serialization issue after 1.8 upgrade

I think you could try StreamExecutionEnvironment.clean(pendingProjectsTag). Oytun Tez 于2019年4月19日 周五下午9:58写道： > Forgot to answer one of your points: the parent class compiles well > without this CEP selector (with timeout signature)... > > > --- > Oytun Tez > > *M O T A W O R D* > The World's

Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )

You could try to set queryable-state.enable to true. And check again. Vishal Santoshi 于2019年4月25日 周四上午1:40写道： > Any one ? > > On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi < > vishal.santo...@gmail.com> wrote: > >> Hello folks, >> >> Following >>

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Congratulations!! Best, Guowei Congxian Qiu 于2019年8月23日周五 上午9:32写道： > Congratulations, and thanks for everyone who make this release possible. > Best, > Congxian > > > Kurt Young 于2019年8月23日周五 上午8:13写道： > >> Great to hear! Thanks Gordon for driving the release, and it's been a >> great

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

Congratulations Zili ! Best, Guowei Fabian Hueske 于2019年9月11日周三 下午7:02写道： > Congrats Zili Chen :-) > > Cheers, Fabian > > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu : > >> Congrats Zili! >> >> Thanks, >> Biao /'bɪ.aʊ/ >> >> >> >> On Wed, 11 Sep 2019 at 18:43, Oytun Tez wrote: >>

Re: why operator not chained?

Hi, Steven 1. The `icebergsink-writer` operator does not chain with the first operator is because the "icebergsink-writer" operator 's ChainingStrategy is NULL or HEAD. You could verify it by printing writer.getChainingStrategy. 2. The two operators use the FORWARD partition if the parallelism of

Re: java.time.LocalDateTime in POJO type

Hi KristoffSC, As far as I know, there is no simple API to let you directly use the LocalTimeTypeInfo for LocalDataTime in your POJO class. (maybe other guys know) If the serializer/deserializer of LocalDataTime is very critical for you there might be two methods. 1. Using the

Re: Reinterpreting a pre-partitioned data stream as keyed stream

Hi, 1. Is the key that is used by the keyBy after point 1 precisely the same as the key used by the 4a and 4b? If yes, I think you could use the reinterpretAsKeyedStream to avoid the shuffle. 2. You could use SingleOutputStreamOperator::setChainingStrategy to disable the chain or use

Re: Reinterpreting a pre-partitioned data stream as keyed stream

Hi, Krzysztof When you use the *reinterpretAsKeyedStream* you must guarantee that partition is the same as Flink does by yourself. But before going any further I think we should know whether normal DataStream API could satisfy your requirements without using *reinterpretAsKeyedStream.* An

Re: DisableGenericTypes is not compatible with Kafka

Hi, I think there could be two workaround ways to 'disableGenericType' in case of KafkaSource : 1. adding the TypeInfo annotation [1] to the KafaTopicPartition. 2. using the reflection to call the private method. :) Maybe we could add this TypeInfo annotation to the KafakaConnector. [1]

Re: 回复：Re: Writing _SUCCESS Files (Streaming and Batch)

Hi, Theo Thank you for sharing your solution. >From your description, it seems that what you need is a listener that could notify the state change of the partition/bucket: created/updated/closed. (maybe you don't need the close notify). I am not familiar with Impala. So what I want to know is why

Re: 回复：Re: Writing _SUCCESS Files (Streaming and Batch)

lelism 1 task, I see that they all worked on the same partition > and submit one query to impala to update this partition. > > Having a notifier sounds great, if it can be executed with parallelism 1 > for all sink tasks.. > > Best regards > Theo > > --

Re: Stream Iterative Matching

Hi, Marc I think the window operator might fulfill your needs. You could find the detailed description here[1] In general, I think you could choose the correct type of window and use the `ProcessWindowFunction` to emit the elements that match the best sum. [1]

Re: CoFlatMap has high back pressure

Hi, Sundar 1. I think you might need to jstack the java process that is a bottleneck and find where the task is stuck. 2. Could you share the code that your job looked like? I think maybe it could help people to know what exactly happened. Best, Guowei sundar 于2020年5月20日周三 上午5:24写道： >

Re: CoFlatMap has high back pressure

Hi Sundar, 1. Could you check the GC status of the process? or you could increase the memory size of your TM. (I find that you use the value state and I assume that you use the MemoryStatebackend) 2. AFAIK there is no performance limitation in using the `connect` operator for mixing the

[DISCUSS] FLIP-143: Unified Sink API

Hi, devs & users As discussed in FLIP-131[1], Flink will deprecate the DataSet API in favor of DataStream API and Table API. Users should be able to use DataStream API to write jobs that support both bounded and unbounded execution modes. However, Flink does not provide a sink API to guarantee

Re: [ANNOUNCE] Apache Flink 1.10.2 released

Hi, Thanks a lot for being the release manager Zhu Zhu! Thanks everyone contributed to this! Best, Guowei On Wed, Aug 26, 2020 at 11:18 AM Yun Tang wrote: > Thanks for Zhu's work to manage this release and everyone who contributed > to this! > > Best, > Yun Tang >

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Hi, Klou Thanks for your proposal. It's a very good idea. Just a little comment about the "Batch vs Streaming Scheduling". In the AUTOMATIC execution mode maybe we could not pick BATCH execution mode even if all sources are bounded. For example some applications would use the

Re: [DISCUSS] FLINK-17989 - java.lang.NoClassDefFoundError org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter

Hi, I think the StreamingFileSink could not support Azure currently. You could find more detailed info from here[1]. [1] https://issues.apache.org/jira/browse/FLINK-17444 Best, Guowei Israel Ekpo 于2020年5月28日周四 上午6:04写道： > You can assign the task to me and I will like to collaborate with

Re: java.lang.NoSuchMethodError while writing to Kafka from Flink

Hi 1. You could check whether the 'org.apache.flink.api.java.clean' is in your classpath first. 2. Do you follow the doc[1] to deploy your local cluster and run some existed examples such as WordCount? [1]

Re: Tumbling windows - increasing checkpoint size over time

the whole time period is consistent (follows > the same pattern). > > > > Changing the checkpoint interval seemed to fix the problem of the large > and growing checkpoint size but I’m not sure why. > > > > Thanks! > > > > -Matt > > > > *From: *Till Rohrm

Re: Tumbling windows - increasing checkpoint size over time

Hi, Matt The total size of the state of the window operator is related to the number of windows. For example if you use keyby+tumblingwindow there would be keys number of windows. Hope this helps. Best, Guowei Wissman, Matt 于2020年5月27日周三 上午3:35写道： > > Hello Flink Community, > > > > I’m running a

Re: Stream Iterative Matching

Hi, Marc 1. I think you should choose which type of window you want to use first. (Thumbling/Sliding/Session) From your description, I think the session window maybe not suit your case because there is no gap. 2. >>> how this would work in practise or how to handle the case where timers fire for

Re: Re: [ANNOUNCE] Apache Flink 1.11.2 released

Thanks Zhuzhu for driving the release!!! Best, Guowei On Fri, Sep 18, 2020 at 5:10 PM Yun Gao wrote: > Great! Very thanks @ZhuZhu for driving this and thanks for all contributed > to the release! > > Best, > Yun > > --Original Mail -- > *Sender:*Jingsong Li >

Re: Date Deserialization probleme

Hi, Maybe I miss something. But I do not get where the problem is. Could you just clarify which part of your result is not expected? Best, Guowei Aissa Elaffani 于2020年6月26日周五 上午6:58写道： > Hello guys, > I want to deserialize a kafka message in JSON format to a POJO, using > Jackso mapper, and I

Re: NoSuchMethodError: org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader

Hi, In 1.10 there is no 'Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader' . So I think there might be a legacy S3FileSystemFactory in your jar. You could check whether there is a 'org.apache.flink.fs.s3presto.common.HadoopConfigLoader' in your jar or not. If there is one you could remove

Re: Reading from AVRO files

org.apache.avro.Conversion[] { TIMESTAMP_CONVERSION, DATE_CONVERSION, TIME_CONVERSION, null }; Best, Guowei Best, Guowei Guowei Ma 于2020年6月11日周四 下午4:12写道： > Hi, > I write a test for the case 1 but it does not throw any exception.

Re: Reading from AVRO files

Hi, for the 1.8.2(case 1) you could try the format.setReuseAvroValue(false); Best, Guowei Lorenzo Nicora 于2020年6月11日周四 下午5:02写道： > Hi Arvid, > > thanks for the point about catching records. Gotcha! > > Sorry I cannot share the full schema or generated code. It's a 3rd party > IP and we signed

Re: Reading from AVRO files

Hi, I write a test for the case 1 but it does not throw any exception. I use the org.apache.flink.formats.avro.generated.JodaTimeRecord for the test. Best, Guowei Arvid Heise 于2020年6月11日周四 下午3:58写道： > Hi Lorenzo, > > I'm glad that it worked out somehow, but I'd still like to understand what >

Re: pause and resume flink stream job based on certain condition

Hi, Eleanore 1. AFAIK I think only the job could "pause" itself. For example the "query" external system could pause when the external system is down. 2. Maybe you could try the "iterate" and send the failed message back to retry if you use the DataStream api. Best, Guowei On Mon, Nov 30,

Re: left join flink stream

gt; > > Best Regards, > > . > > > > > On Mon, 23 Nov 2020 at 14.44 Guowei Ma wrote: > >> Could you share your code? >> Best, >> Guowei >> >> >> On Mon, Nov 23, 2020 at 12:05 PM tkg_cangkul >> wrote: >> >>> Hi

Re: Flink UDF registration from jar at runtime

Hi, Jakub In theory there should not be any problem because you could register the function object. So would you like to share your code and the shell command that you submit your job? Best, Guowei On Mon, Dec 7, 2020 at 3:19 AM Jakub N wrote: > The current setup is: Data in Kafka -> Kafka

Re: How to filter kafka stream from multiples source csv files.

Hi, Amr What sink do you use? I think it means that the sink does not support the "upsert". If you use Kafka as a sink[1] I think you could choose to try it after 1.12. [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/upsert-kafka.html Best, Guowei On Mon, Dec 7,

Re: Flink UDF registration from jar at runtime

{ > > public String eval(String s) { > return "myFunction - " + s; > } > > } > > Execution works as follows: A QueryCommand instance is created, some > properties are being set, *execute()* will be invoked > > Let me know if any other relev

Re: Flink UDF registration from jar at runtime

UDF's from outside the classpath? > > Kind regards, > > Jakub > > ------ > *Von:* Jakub N > *Gesendet:* Montag, 7. Dezember 2020 12:59 > *An:* Guowei Ma > *Cc:* user@flink.apache.org > *Betreff:* Re: Flink UDF registration from jar at runtime > > Hi Gu

Re: Flink UDF registration from jar at runtime

ese explanations are unclear or too vague. > > Kind regards, > > Jakub > > -- > *Von:* Guowei Ma > *Gesendet:* Dienstag, 8. Dezember 2020 06:34 > *An:* Jakub N > *Cc:* user@flink.apache.org > *Betreff:* Re: Flink UDF registration from jar

Re: Adding keyed state to test harness before calling process function.

Hi, Macro I think you could look at testScalingUp() at flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksIncrementalCheckpointRescalingTest.java Best, Guowei On Fri, Nov 13, 2020 at 10:36 AM Marco Villalobos wrote: > Hi, > > I would

Re: Adding keyed state to test harness before calling process function.

Count); > out.collect(newCount); > } > > What I mean by initialize keyed state, is that I want to call > processElement with values already existing in it. > > On Thu, Nov 12, 2020 at 7:52 PM Guowei Ma wrote: > >> Hi, Macro >> I think you could look at testScalingUp() at &g

Re: left join flink stream

Hi, Youzha In general `CoGroup` is for the window based operation. How it could satisfy your requirements depends on your specific scenario. But if you want to look at the mysql table as a dimension table. There might be other two ways: 1. Using Table/Sql SDK. You could find a sql

Re: Random Task executor shutdown

Hi, Arnaud Would you like to share the log of the shutdown task executor? BTW could you check the gc log of the task executor? Best, Guowei On Mon, Nov 16, 2020 at 8:57 PM LINZ, Arnaud wrote: > (reposted with proper subject line -- sorry for the copy/paste) > -Original message- >

Re: left join flink stream

-docs-release-1.11/dev/table/connectors/jdbc.html [3] https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/streaming/joins.html Best, Guowei On Wed, Nov 18, 2020 at 3:05 PM tkg_cangkul wrote: > Hi Guowei Ma, > > Thanks for your reply, > In my case. > I've some d

Re: left join flink stream

on: Generic RAW types > must have a common type information. at > org.apache.flink.table.planner.calcite.FlinkTypeFactory.resolveAllIdenticalTypes(FlinkTypeFactory.scala:381) > > > is there any somethine that i missed here? > > On 23/11/20 08:43, Guowei Ma wrote: > > Hi > One way would look like as following > 1

Re: Initializing broadcast state

cation state from myState > > } > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/co/BroadcastProcessFunction.html > . > > > Best, > Nick. > > On Sun, Jan 24, 2021 at 11:54 PM Guowei Ma wrote: &

Re: Comment in source code of CoGroupedStreams

Hi, I think you could try something like this firstStream .coGroup(secondStream) .where(_.id) .equalTo(_.id) .window(TumblingEventTimeWindows.of(Time.seconds(1))) .with(new MyCogroupFunction()) .uid("myCoGroup") Best, Guowei On Fri, Jan 22, 2021 at 4:33 AM

Re: Where should a secondary flow for late events processing be defined?

Hi, Jose What I understand your question is Your job has two stages. You want to handle the first stage differently according to the event time of the Stream A. It means that if the event time of Stream A is “too late” then you would enrich Stream A with the external system and or you would

Re: Initializing broadcast state

Hi, Nick You might need to handle it yourself If you have to process an element only after you get the broadcast state. For example, you could “cache” the element to the state and handle it when the element from the broadcast side elements are arrived. Specially if you are using the

Re: Initializing broadcast state

ps://stackoverflow.com/questions/58307154/initialize-the-content-of-a-mapstate > > > Best, > Nick > On Sun, Jan 24, 2021 at 9:23 PM Guowei Ma wrote: > >> Hi, Nick >> You might need to handle it yourself If you have to process an element >> only after you get

Re: Unable to query/print the incomplete bucket state

Hi, Falak >>>Now if I try to query this bucket (state) using a queryable state, then i don't get the results. AFAIK, Flink does not have a way to let user query the state of the `WiindowOperator`. It needs to expose the window operator's internal implementation, which might be difficult to

Re: [ANNOUNCE] Apache Flink 1.12.1 released

Thanks Xintong's effort! Best, Guowei On Tue, Jan 19, 2021 at 5:37 PM Yangze Guo wrote: > Thanks Xintong for the great work! > > Best, > Yangze Guo > > On Tue, Jan 19, 2021 at 4:47 PM Till Rohrmann > wrote: > > > > Thanks a lot for driving this release Xintong. This was indeed a release >

Re: Initializing broadcast state

ache elements. > Thanks a ton for your help. > > Best, > Nick > > On Mon, Jan 25, 2021 at 10:38 PM Guowei Ma wrote: > >> Hi,Nick >> I do not think you could update the `myState` in the >> `processBroadcastElement`. It is because you need a key before

Re: Discard checkpoint files through a single recursive call

hi, Jiang I am afraid of misunderstanding what you mean, so can you elaborate on how you want to change it? For example, which interface or class do you want to add a method to? Although I am not a state expert, as far as I know, due to incremental checkpoints, when CompleteCheckpoint is

Re: Got multiple issues when running the tutorial project "table-walkthrough" on IDEA

Hi, Lingfeng These job errors you posted happened when the job(`SpendReport`) was running on the IDE? According to my understanding, this document[1] & repository[2] mean that the example is to be run in docker, not in IDE. [1]

Re: Checkpoint loading failure

Hi Padarn Will there be these errors if the jobgraph is not modified? In addition, is this error stack all? Is it possible that other errors caused the stream to be closed? Best, Guowei On Tue, Jun 15, 2021 at 9:54 PM Padarn Wilson wrote: > Hi all, > > We have a job that has a medium size

Re: DataStream API in Batch Execution mode

Hi, Macro I think you could try the `FileSource` and you could find an example from [1]. The `FileSource` would scan the file under the given directory recursively. Would you mind opening an issue for lacking the document? [1]

Re: Flink State Processor API Example - Java

Hi Sandeep What I understand is that you want to manipulate the state. So I think you could use the old schema to read the state first, and then write it to a new schema, instead of using a new schema to read an old schema format data. In addition, I would like to ask, if you want to do "State

Re: Flink checkpoint periodically fail

Hi Qihua It seems that the job fail because of checkpoint timeout(10min) from the second picture. I found that the checkpoint fail is because one of your own customs source could not acknowledge the cp. So I think you could add some log in your source to figure out what is happening at the moment.

Re: Metric for JVM Overhaed

Hi Pranjul There are already some system metrics that track the jvm status(CPU/Memory/Threads/GC). You could find them in the [1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#system-metrics Best, Guowei On Fri, Jun 25, 2021 at 2:33 PM Pranjul Ahuja wrote:

Re: Flink 1.4.1 randomically responds HTTP 500 when sending job to Job Manager

Hi Burcu Could you show more logs? I could try to help find out what is happening. But to be honest the 1.4 is too old a version that the community does not support. You’d better upgrade to a newer version. Best, Guowei On Fri, Jun 25, 2021 at 2:48 PM Burcu Gül POLAT EĞRİ wrote: > Dear All, >

Re: Root Exception can not be shown on Web UI in Flink 1.13.0

Hi, Gary I think it might be a bug. So would you like to open a jira for this. And could you share the exception ,which the TaskManagerLocation is null? It might be very helpful to verify the cause. Best, Guowei On Thu, May 13, 2021 at 10:36 AM Yangze Guo wrote: > Hi, it seems to be related

Re: docker based taskmanager can't connect to job/resource manager

Hi, I do not try it. But from the documentation[1] it seems that you might need add the "jobmanager.rpc.address: jobmanager" to the FLINK_PROPERTIES before creating a network. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/ Best,

Re: Flink sql task failure recovery does not work.

Hi I think you could configure some restart strategy[1] likes restart-strategy: fixed-delay [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/task_failure_recovery/#fixed-delay-restart-strategy Best, Guowei On Thu, May 13, 2021 at 12:02 PM 1095193...@qq.com

Re: docker based taskmanager can't connect to job/resource manager

for your reply! This information was still missing. The presenter > mentioned the documentation but I hadn't found it. So your link to the > specific place is valuable too. > > Günter > On 13.05.21 06:09, Guowei Ma wrote: > > Hi, > I do not try it. But from the documentation[

Re: Need Clarity about Checkpoint for Flink-1.12.2

Hi Sudhansu, I think you do not need to set the config in flink-conf. Best, Guowei On Thu, May 13, 2021 at 1:06 PM sudhansu jena wrote: > Hi Team, > > We have recently enabled Check Pointing in our flink job using > FSStateBackend pointing to S3 bucket. > > Below is the sample code for

Re: How does JobManager terminate dangling task manager

Hi, In fact, not only JobManager(ResoruceManager) will kill TimeOut's TaskManager, but if TaskManager finds that it cannot connect to JobManager(ResourceManager), it will also exit by itself. You can look at the time period during which the HB timeout occurred and what happened in the log. Under

Re: Task not serializable when logging in a trait method

Hi, Yik San You need to change the following line: protected final val LOG = LoggerFactory.getLogger(getClass) protected *static* final val LOG = LoggerFactory.getLogger(getClass) Best, Guowei On Mon, May 24, 2021 at 2:41 PM Yik San Chan wrote: > Hi community, > > I have a job that

Re: count the amount of data successfully processed by flink

Hi I think you are right that the metrics are reset after the job restart. It is because the metrics are only stored in the memory. I think you could store the metrics to the Flink's state[1], which could be restored after the job restarted. [1]

Re: How do you debug a DataStream flat join on common window?

Hi, Would you like to share your code? It is very helpful to verify the problem. I think you could use the `JoinedStream.with().uid(xxx)` to set the name/UID . Best, Guowei On Mon, May 24, 2021 at 2:36 PM Marco Villalobos wrote: > Hi, > > Stream one has one element. > Stream two has 2

Re: Kubernetes Application Cluster Not Working

Hi, M Could you give the full stack? This might not be the root cause. Best, Guowei On Wed, Mar 24, 2021 at 2:46 AM Claude M wrote: > Hello, > > I'm trying to setup Flink in Kubernetes using the Application Mode as > described here: >

Re: Fault Tolerance with RocksDBStateBackend

Hi, You need some persistent storages(like hdfs) for the checkpoint. It is Flink's fault tolerance prerequisites.[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/state/checkpointing.html#prerequisites Best, Guowei On Wed, Mar 24, 2021 at 1:21 PM Maminspapin

Re: About Memory Spilling to Disk in Flink

Hi, Roc Could you explain more about your question? Best, Guowei On Wed, Mar 24, 2021 at 2:47 PM Roc Marshal wrote: > Hi, > > Can someone tell me where flink uses memory spilling to write to disk? > Thank you. > > Best, Roc. > > > >

Re: Re: Re: [DISCUSSION] Introduce a separated memory pool for the TM merge shuffle

>> temporarily using taskmanager.memory.network.batch-shuffle-read.size in >> my PR now. Any suggestions about that? >> >> Best, >> Yingjie (Kevin) >> >> -- >> 发件人：Guowei Ma >> 日 期：2021年03月09日 17:28:35 >>

Re: Native kubernetes execution and History server

Hi, After some discussion with Wang Yang offline, it seems that there might be a jobmanager failover. So would you like to share full jobmanager log? Best, Guowei On Wed, Mar 24, 2021 at 10:04 PM Lukáš Drbal wrote: > Hi, > > I would like to use native kubernetes execution [1] for one batch job

Re: flink sql jmh failure

Hi, I am not an expert of JMH but it seems that it is not an error. From the log it looks like that the job is not finished. The data source continues to read data when JMH finishes. Thread[Legacy Source Thread - Source: TableSourceScan(table=[[default_catalog, default_database,

Re: Native kubernetes execution and History server

NK_HOME}/plugins/s3-fs-presto > COPY flink-s3-fs-presto-1.12.2.jar $FLINK_HOME/plugins/s3-fs-presto/ > > Thanks! > > On Thu, Mar 25, 2021 at 9:24 AM Guowei Ma wrote: > >> Hi, >> After some discussion with Wang Yang offline, it seems that there might >> be a job

Re: [DISCUSS] Feature freeze date for 1.13

Hi, community: Friendly reminder that today (3.31) is the last day of feature development. Under normal circumstances, you will not be able to submit new features from tomorrow (4.1). Tomorrow we will create 1.13.0-rc0 for testing, welcome to help test together. After the test is relatively

Re: Checkpoint timeouts at times of high load

Hi, I think there are many reasons that could lead to the checkpoint timeout. Would you like to share some detailed information of checkpoint? For example, the detailed checkpoint information from the web.[1] And which Flink version do you use? [1]

Re: s3 FileSystem Error "s3 file system implementation does not support recoverable writers"

nt.java:5062) > ~[?:?] > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008) > ~?:?] > at > com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3581) > ~[?:?] > at > org.apache.hadoop.fs.s3a.S3A

Re: Flink Taskmanager failure recovery and large state

cksDBStateBackend{checkpointStreamBackend=File State > Backend (checkpoints: 'gs:///flink-checkpoints', savepoints: > 'gs:///flink-savepoints', asynchronous: TRUE, fileStateThreshold: > 1048576), localRocksDbDirectories=[/rocksdb], > enableIncrementalCheckpointing=TRUE, numberOfTransferT

Re: [DISCUSS] Feature freeze date for 1.13

it's a >>>> useful improvement to sql client and won't affect >>>> other components too much. We were plan to merge it yesterday but met >>>> some tricky multi-process issue which >>>> has a very high possibility hanging the tests. It took us a while

Re: s3 FileSystem Error "s3 file system implementation does not support recoverable writers"

Hi, Robert I think you could try to change the "s3://argo-artifacts/" to " s3a://argo-artifacts/". It is because that currently `StreamingFileSink` only supports Hadoop based s3 but not Presto based s3. [1] [1]

  1   2   >