Re: Conditional multi collect in flink

Hi Tauseef, It sounds like you want to use side outputs (via a ProcessFunction, versus a FlatMap). — Ken > On Dec 4, 2023, at 8:23 AM, Tauseef Janvekar > wrote: > > Dear Team, > > I was wonderin

Re: [ANNOUNCE] Apache Flink Kafka Connectors 3.0.2 released

Hi Gordon, Thanks for the release! I've pushed one hotfix [1], to make sure that the Flink documentation shows the correct version number for the Flink version it's compatible with. Best regards, Martijn [1] https://github.com/apache/flink-connector-kafka/commit/6c3d3d06689336f2fd37bfa5a3b17a5

Re: failed when job graph change

Hi restart the job it's ok and i do that , but i must cancel the job and submit a new one and i dont want the data from the state forget to mention that i use the parameter "-allowNonRestoredState" my steps: 1. stop the job with savepoint 2. run the updated job ( update job graph) from savepoin

Re: Exploring Data Storage Mechanisms in Flink with RocksDB: A Focus on Windowing and Aggregation

Thank you for providing the details. Can it be confirmed that the Hashmap within the accumulator stores the map in RocksDB as a binary object and undergoes deserialization/serialization during the execution of the aggregate function? Thanks, Arjun On Mon, 4 Dec 2023 at 12:24, Xuyang wrote: > Hi

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

Hi guys, Forking in sbt solved the issue (Test / fork := true). On Sun, Dec 3, 2023 at 7:48 AM Barak Ben-Nathan wrote: > By the way, I also upgraded to flink-connector-kafka ver. 3.0.2-1.18, to > no avail. > > On Sun, Dec 3, 2023 at 7:45 AM Barak Ben-Nathan > wrote: > >> Thank's Jim, >> >> Un

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

By the way, I also upgraded to flink-connector-kafka ver. 3.0.2-1.18, to no avail. On Sun, Dec 3, 2023 at 7:45 AM Barak Ben-Nathan wrote: > Thank's Jim, > > Unfortunately, this did not resolve the issue. > > I tried downgrading to 17.2 -- and everything works fine. > In version 18.0 -- I also ad

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

Thank's Jim, Unfortunately, this did not resolve the issue. I tried downgrading to 17.2 -- and everything works fine. In version 18.0 -- I also added flink-clients (in addition to flink-core), but the problem persists On Fri, Dec 1, 2023 at 10:44 PM Jim Hughes via user wrote: > Hi Barak,

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

Hi Barak, The missing class is in "flink-core", I think adding that dependency will provide it. The release notes for 1.14 note that Connectors no longer bundle "flink-core". I imagine this is what has caused your issue. https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/fl

Re: Flink Kubernetes Operator: Why one Helm repo for each version?

Ok, thanks for the clarification. On Fri, Dec 1, 2023 at 2:05 PM Gyula Fóra wrote: > Hi! > > I already answered your question on slack : > > > “The main reason is that this allows us to completely separate release > resources etc. much easier for the release process > > But this could be improve

Re: Flink Kubernetes Operator: Why one Helm repo for each version?

Hi! I already answered your question on slack : “The main reason is that this allows us to completely separate release resources etc. much easier for the release process But this could be improved in the future if there is a good proposal for the process” Please do not cross post questions bet

Re: Profiling on flink jobs

I missed the Reply All button in my previous message. Here's my previous email for the sake of transparency sent to the user ML once more: Hi Oscar, sorry for the late reply. I didn't see that you posted the question at the beginning of the month already. I used jmap [1] in the past to get some s

Re: [EXTERNAL] Re:Flink SQL and createRemoteEnvironment

Hi Xuyang, thanks for the reply. I've listed a complete example below. I have a working flink cluster running in AWS and accepting connections over https. I think I'm probably just very confused about how this is meant to work. What I expected:   - The executeSql statements to be executed r

Re: How to call the open method of JdbcSink?

Hi Feng, Thank you for your response and the suggestion. I was able to cast the SinkFunction to GenericJdbcSinkFunction which enabled me to override the open and close methods. On proceeding further, I have observed that if the batch encounters BatchUpdateException due to a packet inside the batch

Re: [EXTERNAL] TaskManagers Crushing

Were you ever able to find a workaround for this? I also have transient failures due to org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException. From: Kenan Kılıçtepe Sent: Saturday, August 19, 2023 5:50 PM To: user@flink.apache.org Subject:

Re: Doubts about state and table API

Hi Oscar, could you provide the Java code to illustrate what you were doing? The difference between version A and B might be especially helpful. I assume you already looked into the FAQ about operator IDs [1]? Adding the JM and TM logs might help as well to investigate the issue, as Yu Chen mentio

Re: Java 17 as default

The 1.18 Docker images were pushed on Oct 31. This also included Java 17 images [1]. [1] https://hub.docker.com/_/flink/tags?page=1&name=java17 On Wed, Nov 15, 2023 at 7:56 AM Tauseef Janvekar wrote: > Dear Team, > > I saw the documentation for 1.18 and Java 17 is not supported and the > image

RE: Re: Re: inputQueueSize metric

Thank you for your previous response Feng. It clears things up a little. Regarding the inputQueueSize metric. What does it express after all? Observing the inputQueueSize and inPoolUsage metrics in Prometheus, I found that when the inPoolUsage is high, the inputQueueSize is low. This observati

Re: [ANNOUNCE] Apache Flink 1.17.2 released

Thanks Yun for driving the release. Thanks a lot to everyone that has contributed with bug fixes and other improvements! Best, Leonard > 2023年11月29日 下午1:05，Yun Tang 写道： > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.17.2, which is the second bugfi

Re: Re: inputQueueSize metric

Hi Dimitris Yes, the inputQueueLength metric can only represent the number of buffers, and cannot obtain the actual size of the messages. Regarding the inputQueueSize metric, it ignores the LocalChannel. Therefore, when tasks are connected through the Forward partition strategy, it will consisten

Re: How to call the open method of JdbcSink?

Hi Sai I think you can directly cast SinkFunction to GenericJdbcSinkFunction. https://github.com/apache/flink-connector-jdbc/blob/b477d452ba3aac38d53d1f5d4c4820bdad3ad9cd/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/JdbcSink.java#L63C41-L63C41 ``` public static SinkFun

Re: Flink SQL and createRemoteEnvironment

Hi, createRemoteEnvironment(...) methods have some obscure documentation. createRemoteEnvironment(String host,int port,String ... jarF

Re: Getting a list of tasks for a running job

Hi Yuxin, Added flink user group Thanks, Tauseef On Tue, 28 Nov 2023 at 11:38, Tauseef Janvekar wrote: > Hi Yuxin, > We have deployed it on kubernetes using helm chart - > https://github.com/bitnami/charts/blob/main/bitnami/flink/values.yaml > We have used ingress and enabled basic authenticat

Re: Job Manager and Task Manager unable to communicate

Adding flink user group On Tue, 28 Nov 2023 at 13:39, Tauseef Janvekar wrote: > Did you set some specific job manager or task manager deployment > parameters ? - No > > Did you test without the basic ingress auth ? to be sure this is not > related to that. - yes we did. And the problem persists.

Re: Checkpoint RMM

Hi Oscar, > but we don't understand why this incremental checkpoint keeps increasing AFAIK, when performing incremental checkpoint, the RocksDBStateBackend will upload the new created SST files to remote storage. The total size of these files is the incremental checkpoint size. However, the new c

Re:回复: oomkill issue

Hi, Prashant. I think Yu Chen has given a professional troubleshooting ideas. Another thing I want to ask is whether you use some user defined function to store some objects? You can firstly dump the memory and get more details to check for memory leaks. -- Best！ Xuyang 在 2023-1

Re: Getting a list of tasks for a running job

Hi, Tauseef, AFAIK, the most common way to get a list of tasks that a particular job executes is through Flink's Web UI or REST API. Using the Flink Web UI: When you run a Flink cluster, a Web UI is launched by default on port 8081 of the JobManager. By accessing this Web UI through a browser, y

Re: Error flink 1.18 not found ExecutionConfig

it be problems with dependencies or > execution options? > > Thanks, > Dulce Morim > > -Original Message- > From: Yunfeng Zhou > Sent: 27 de novembro de 2023 01:26 > To: Dulce Morim > Subject: Re: Error flink 1.18 not found ExecutionConfig > > CAUTION:

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

Thanks! Yeah I am not sure why it's handled so different with non-native k8s mode. If it's possible I think this would be a huge improvement. On Mon, Nov 20, 2023, 12:55 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Trystan, I'm actually not very familiar with the operator's i

RE: Re: inputQueueSize metric

As per the docs, the `inputQueueLength` metric refers to the number of queued input buffers, and cannot be used on its own in order to determine buffered records. For instance, if I know that there are 5 queued input buffers, I cannot conclude anything regarding buffered records if the size of

Re: inputQueueSize metric

Hi Dimitris Maybe you can use the `inputQueueLength` metric. Best, Feng On Tue, Nov 28, 2023 at 12:07 AM Dimitris Mpanelas via user < user@flink.apache.org> wrote: > Hello, > > I am trying to determine the buffered records in the input buffers of a > task. I found the inputQueueSize metric. Ac

Re: Metrics not available

Hi Oscar Did you set state.backend.latency-track.keyed-state-enabled=true; https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/config/#state-backend-latency-track-keyed-state-enabled Best, Feng On Mon, Nov 27, 2023 at 8:42 PM Oscar Perez via user wrote: > Hi, > > We ar

Re: Job Manager and Task Manager unable to communicate

Hello, Tauseef, Can you give more details ? Are your task manager and job manager on the same vm ? How did you configure the Job manager address in the task manager conf file ? Did you modify the binding in configuration files ? Benoit Le lun. 27 nov. 2023 à 14:29, Tauseef Janvekar a écrit :

Re: Flink 1.18 and java 17

Hi Lasse, The default flink-conf.yaml file bundled in the distribution should already have a preset env.java.opts.all config for Java 17. Have you tried that? Best, Zhanghao Chen From: Lasse Nedergaard Sent: Monday, November 27, 2023 21:20 To: user Subject: Fli

Re: Elasticsearch connector support?

Hi HangThanks for the link. I will wait for 3.1 connector release and hope it will be included. Med venlig hilsen / Best regardsLasse NedergaardDen 27. nov. 2023 kl. 12.00 skrev Hang Ruan :Hi, Lasse.There is already a discussion about the connector releases for 1.18[1].Best,Hang[1] https://lists.a

Re: Elasticsearch connector support?

Hi, Lasse. There is already a discussion about the connector releases for 1.18[1]. Best, Hang [1] https://lists.apache.org/thread/r31f988m57rtjy4s75030pzwrlqybpq2 Lasse Nedergaard 于2023年11月24日周五 22:57写道： > Hi > > From the documentation I can see there isn’t any ES support in Flink 1.18 > righ

Re: 退订

Hi Jintao, Please send an email to user-unsubscr...@flink.apache.org to unsubscribe the user mailing list. Jintao Ma 于2023年11月27日周一 09:24写道： > 退订 >

Re: Operator ids

It is not supported yet. Curious why do you need to get the operator IDs? They are usually only used internally. Best, Zhanghao Chen From: rania duni Sent: Saturday, November 25, 2023 20:44 To: user@flink.apache.org Subject: Operator ids Hello! I would like to

Re: dependency error with latest Kafka connector

No problem, glad to hear that it's working now! With release candidates, we always publish the url for staged artifacts in the release candidate vote threads so that you can point your code to compile against those for testing purposes. Would be great to have your +1 on the vote thread for 3.0.2

Re: dependency error with latest Kafka connector

Thanks Gordon! I didn't know the name of the repository https://repository.apache.org/content/repositories/orgapacheflink-1675/ Additionally something learned. Yes, with the new version I can add the dependency "org.apache.flink" % "flink-connector-kafka" % "3.0.2-1.18", and compile it without

Re: dependency error with latest Kafka connector

Hi Günter, With Maven you'd list the staged repository holding the RC artifacts as a repository: ``` test_kafka_rc Apache Flink Kafka Connector v3.0.2 https://repository.apache.org/content/repositories/orgapacheflink-1675/ ``` With SBT, I think the equivalent is usi

Re: Operator ids

You, unfortunately, just cant AFAIK On Sat, 25 Nov 2023 at 14:45, rania duni wrote: > Hello! > > I would like to know how can I get the operator ids of a running job. I > know how can I get the task ids but I want the operator ids! I couldn’t > find something to the REST API docs. > Thank you.

Re: dependency error with latest Kafka connector

Hi Gordon, thanks for working on it. How can I reference the repository for the new artifact. Referencing 3.0.2-18 I get an unresolved dependency error. Thanks for a hint. Günter sbt:flink_essential_swrapper> compile [info] Updating [info] Resolved  dependencies [warn] [warn]     Note: Unre

Re: [EXTERNAL] Re: flink s3[parquet] -> s3[iceberg]

start instances of the job through the job api. > > Unless reading the parquet to a temporary table doesn’t need the schema > definition? I couldn't really work things out from the links. > > Dan > -- > *From:* Feng Jin > *Sent:* Thursday, Nove

Re: dependency error with latest Kafka connector

Hi all, I've cherry-picked FLINK-30400 onto v3.0 branch of flink-connector-kafka. Treating this thread as justification to start a vote for 3.0.2 RC #1 immediately so we can get out a new release ASAP. Please see the vote thread here [1]. @guenterh.lists Would you be able to test this RC and se

Re: dependency error with latest Kafka connector

> built a fat uber jar from quickstart with Flink 1.18.0 for > flink-streaming-java and flink-clients, and flink-connector-kafka version > 3.0.1-1.18 > then submitted to local Flink cluster 1.18.0. Things worked as expected and > the job ran fine. Hey，@Gordan I guess things may work as expecte

Re: Confluent Kafka conection error

Hi Hang, Few more points regarding this issue 1. The issue does not replicate in my local installation and occurs only on kubernetes server installation. 2. We have used kubernetes operator to install flink on the server Please let me know if any other info is required here. Thanks, Tauseef On

Re: Confluent Kafka conection error

Hi Hang, I cross checked this issue multiple times. I also upgraded to flink 1.18 but the issue persists. Can you please let me know a few guidelines on how to investigate this and fix it positively. Thanks, Tauseef On Thu, 23 Nov 2023 at 18:08, Tauseef Janvekar wrote: > Thanks Hang. > > I go

Re: dependency error with latest Kafka connector

r POM. > The *-1.18 prefix on the externalized connector artifacts simply indicate > that they are guaranteed to be compilable and compatible with Flink 1.18.x > dependencies. > > As a sanity check, I’ve also re-done the tests that were validated during the > 3.0.1 release process to se

Re: dependency error with latest Kafka connector

Flink 1.18.x dependencies. As a sanity check, I’ve also re-done the tests that were validated during the 3.0.1 release process to see why any issues slipped, but it turns out to be working fine: - built a fat uber jar from quickstart with Flink 1.18.0 for flink-streaming-java and flink-clients

Re: dependency error with latest Kafka connector

Hi all, There seems to be an issue with the connector release scripts used in the release process that doesn't correctly overwrite the flink.version property in POMs. I'll kick off a new release for 3.0.2 shortly to address this. Sorry for overlooking this during the previous release. Best, Gord

Re: [EXTERNAL] Re: flink s3[parquet] -> s3[iceberg]

Dan From: Feng Jin Sent: Thursday, November 23, 2023 6:49:11 PM To: Oxlade, Dan Cc: user@flink.apache.org Subject: [EXTERNAL] Re: flink s3[parquet] -> s3[iceberg] Hi Oxlade I think using Flink SQL can conveniently fulfill your requirements. For S3 Parquet files, you can create a t

Re: flink s3[parquet] -> s3[iceberg]

Hi Oxlade I think using Flink SQL can conveniently fulfill your requirements. For S3 Parquet files, you can create a temporary table using a filesystem connector[1] . For Iceberg tables, FlinkSQL can easily integrate with the Iceberg catalog[2]. Therefore, you can use Flink SQL to export S3 file

Re: dependency error with latest Kafka connector

Hi Danny thanks for taking a look into it and for the hint. Your assumption is correct - It compiles when the base connector is excluded. In sbt: "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18" exclude("org.apache.flink", "flink-connector-base"), Günter On 23.11.23 14:24, Dan

Re: dependency error with latest Kafka connector

Hey all, I believe this is because of FLINK-30400. Looking at the pom I cannot see any other dependencies that would cause a problem. To workaround this, can you try to remove that dependency from your build? org.apache.flink flink-connector-kafka 3.0.1-1.18

Re: dependency error with latest Kafka connector

Hi, Gurnterh It seems a bug for me that 3.0.1-1.18 flink Kafka connector use flink 1.17 dependency which lead to your issue. I guess we need propose a new release for Kafka connector for fix this issue. CC: Gordan, Danny, Martijn Best, Leonard > 2023年11月14日 下午6:53，Alexey Novakov via user

Re: Confluent Kafka conection error

Thanks Hang. I got it now. I will check on this and get back to you. Thanks, Tauseef. On Thu, 23 Nov 2023 at 17:29, Hang Ruan wrote: > Hi, Tauseef. > > This error is not that you can not access the Kafka cluster. Actually, > this error means that the JM cannot access its TM. > Have you ever ch

Re: Confluent Kafka conection error

Hi, Tauseef. This error is not that you can not access the Kafka cluster. Actually, this error means that the JM cannot access its TM. Have you ever checked whether the JM is able to access the TM? Best, Hang Tauseef Janvekar 于2023年11月23日周四 16:04写道： > Dear Team, > > We are facing the below iss

Re: [PyFlink] Collect multiple elements in CoProcessFunction

Hi David, Thanks for the confirmation. Let's fix the docs: https://github.com/apache/flink/pull/23776 Thanks, Alex On Sun, 19 Nov 2023 at 01:55, David Anderson wrote: > Hi, Alex! > > Yes, in PyFlink the various flatmap and process functions are implemented > as generator functions, so they us

Re: Is RestClusterClient recommended?

Don't use the RestClusterClient; you can generate one from the openapi spec (see the docs). On 16/11/2023 20:36, Adrian Alexandru Vasiliu wrote: Hello, For a programmatic use in Java of the Flink REST API, which of these options would be the best choice? 1. Direct use via a REST client 2.

Re:Re: Re: flatmap returns a custom class object

Hi, Tauseef. This error happens after the job is running, so I think at least it proves that there is no problem with the modified code. Now let's focus on the new error. The exception seems that flink can't read the metadata from kafka to get all of the partitions about this topic. Can you ch

Re: Increasing maximum number of FlinkDeployments that the Operator can handle

brain scenarios will also help as we can re-enable HA and have operator pods running in active-passive mode. On Wed, Nov 8, 2023 at 2:59 PM Tony Chen wrote: > Currently, 16GB of heap size is allocated to the flink-kubernetes-operator > container by setting *jvmArgs.operator*, and this didn&#x

Re: [DISCUSS][FLINK-32993] Datagen connector handles length-constrained fields according to the schema definition by default

Thanks Yubin and Jane for the discussion! +1 to fix this bug, although it's usually used as a test source, it's important to provide the correct behavior for users. for the invalid field length configured by users, I think it's better to raise an error instead of using default value silently. ta

Re: [DISCUSS][FLINK-32993] Datagen connector handles length-constrained fields according to the schema definition by default

Hi Yubin, Thanks for driving this discussion. Perhaps a specific example can better illustrate the current issue. Considering the following DDL, f0 will always be generated with a default char length of 100, regardless of char(5), bcause the connector option 'fields.f0.length' is not specified [1

Re: Flink Kubernetes operator keeps reporting REST client timeout.

Seems the operator didn't get restarted automatically after the configmap is changed. After a roll-out restart, the exception disappeared. Never mind this issue. Thanks. On Tue, Nov 21, 2023 at 11:31 AM Xiaolong Wang wrote: > Hi, > > Recently I upgraded the flink-kubernetes-operator from 1.4.0 t

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

Hi Trystan, I'm actually not very familiar with the operator's internals, but I'd guess that limitation is in Flink itself - application mode is a feature from core Flink, the operator just configures it based on the CRDs it defines. Maybe one of the maintainers can confirm. Regards, Alexis. On M

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

Thanks Alexis, I can give that a try. However, that seems less than ideal from the user's perspective. Is there a technical reason why the operator can't support this combination of modes? I'd really like to just let the system do its thing rather than build a complicated two-jar approach. Thanks

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi Junrui, Thanks for the clarification. On one hand, adding more methods into the RuntimeContext flat will increase the effort for users who will use RuntimeContext. But the impact is limited. It is fine. The big impact is, on the other hand, for users who want to focus on the execution config, t

Re: Questions about the client synchronously obtaining task execution results

Sorry I forgot to add the version information, the version is 1.17 刘峻池 于2023年11月20日周一 13:59写道： > Hi Flink Community > > When I run this command `flink run-application -t yarn-application -sae > mainClass somejar` to submit some batch-task on YARN with Application > Mode, my shell client always

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi Jing, Thank you for your feedback. I understand your concerns regarding putting all methods into the RuntimeContext flat. I would like to share some of my thoughts on this matter. Firstly, this FLIP only proposes the addition of three additional methods, which should not impose too much extra

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

Hi David and Mason, Thanks for your feedback! To David: > Given that the new default feels more complex than the current behavior, if we decide to do this I think it will be important to include the rationale you've shared in the documentation. Sounds make sense to me, I will add the related do

Re: The generated schema is not correct when using filesystem connector and avro format

Hi, julia. I have read the code about this part. The problem as you said is that the RowType passed to the avro-confluent format is nullable, which will cause union with null in the schema. I think FLINK-30438 is the same problem as yours. But I find the RowType passed to avro-confluent format in

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi Junrui, Thanks for bringing this to our attention. First of all, it makes sense to deprecate RuntimeContext#getExecutionConfig. Afaic, this is an issue of how we design API with clean concepts/aspects. There are two issues mentioned in the FLIP: 1. short of user-facing abstraction - we just e

Re: [PyFlink] Collect multiple elements in CoProcessFunction

Hi, Alex! Yes, in PyFlink the various flatmap and process functions are implemented as generator functions, so they use yield to emit results. David On Tue, Nov 7, 2023 at 1:16 PM Alexander Fedulov < alexander.fedu...@gmail.com> wrote: > Java ProcessFunction API defines a clear way to collect d

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hello Wencong, Thank you for your valuable feedback and suggestions. I want to clarify that reviewing existing methods in the ExecutionConfig is not directly related to the proposal in this FLIP. The main focus of this FLIP is to deprecate the specific method RuntimeContext#getExecutionConfig(). I

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

Hi Rui, I suppose we could do some benchmarking on what works well for the resource providers that Flink relies on e.g. Kubernetes. Based on conferences and blogs, it seems most people are relying on Kubernetes to deploy Flink and the restart strategy has a large dependency on how well Kubernetes

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

Hi Trystan, I imagine you can create 2 jars, one should only have a class with the main method, and the other should be a fat jar with everything else for your job. If you create a custom image where your fat jar is placed under /opt/flink/lib/ then I think it would "just work" when specifying the

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

Rui, I don't have any direct experience with this topic, but given the motivation you shared, the proposal makes sense to me. Given that the new default feels more complex than the current behavior, if we decide to do this I think it will be important to include the rationale you've shared in the

Re: dependency error with latest Kafka connector

Same behaviour and assumption on my side Alexey. Thanks for testing it. Günter On 17.11.23 09:40, Alexey Novakov via user wrote: I would expect *flink-connector-kafka:3.0.1-1.18* pointing to *org.apache.flink:flink-connector-base:1.18.0* not to *1.17.0* However, SBT compiles my project ok usin

Re: dependency error with latest Kafka connector

I would expect *flink-connector-kafka:3.0.1-1.18* pointing to *org.apache.flink:flink-connector-base:1.18.0* not to *1.17.0* However, SBT compiles my project ok using such versioning: val flinkVersion = "1.18.0" val flinkVersion17 = "1.17.1" val flinkDependencies = Seq( "org.flinkextended" %% "f

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Sorry, I read something wrong when I read FLIP. It's fine for me. Best, Rui On Thu, Nov 16, 2023 at 2:32 PM Junrui Lee wrote: > Hi Rui, > > Thank you for your feedback. Regarding your question, I want to let you > know that we have included the alternative getter method "@PublicEvolving > Map

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi Rui, Thank you for your feedback. Regarding your question, I want to let you know that we have included the alternative getter method "@PublicEvolving Map getGlobalJobParameters()" in this FLIP. And this method will provide users with global job parameters. I would appreciate it if you could sh

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Thanks Junrui for driving this proposal! Overall looks good to me! I have a question: How do the flink users to getGlobalJobParameters if RuntimeContext#getExecutionConfig is removed? Is there any alternative api? Or is this feature removed? Maybe this is a noob question and I may have missed som

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Thanks Junrui for creating the FLIP and kicking off this discussion. Exposing a mutable ExecutionConfig which is even shared by multiple operators is truly a defect which can result in weird results. +1 Thanks, Zhu Junrui Lee 于2023年11月15日周三 16:53写道： > Hi all, > > I'd like to start a discussio

Re: Disable flink old checkpoint clean

Hi Jinzhong, Sorry to answer you just now. We have switched from incremental checkpoint to non-incremental checkpoint before, I think one of the reasons is the difficulty to handle properly the clean up of checkpoints on S3. But with the flink operator's periodic savepoint it may change. I&

Re: Flink operator autoscaler scaling down

Thanks Maximilian and Gyula, I'll keep you updated. Best, Yang On Sat, 11 Nov 2023 at 16:18, Maximilian Michels wrote: > Hi Yang, > > We're always open to changes / additions to the autoscaler logic and > metric collection. Preferably, we change these directly in the > autoscaler implementation

Re: Elasticsearch source

Hi Tauseef, AFAIK, Flink does not support ElasticSearch as a source connector. It only supports ElasticSearch as a sink connector. Best regards, Junrui Tauseef Janvekar 于2023年11月15日周三 14:36写道： > Dear Team, > > We were looking for some elasticsearch source connector for flink and I > could not

Re: unsubscribe

Hi, Ralph, Please send an email to user-unsubscr...@flink.apache.org to unsubscribe the user mailing list. [1] [1] https://flink.apache.org/what-is-flink/community/ Best, Yuxin Ralph Matthias Debusmann 于2023年11月15日周三 06:18写道： > >

Re: unsubscribe

Hi, Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org. Ralph Matthias Debusmann 于2023年11月15日周三 07:29写道： > >

Re: Kafka Topic Permissions Failure

Hey Hang, We may have encountered a bug. It seems like deleting the job and redeploying "fixed" the issue. We will work on recreating this scenario and will follow-up at some point. On Mon, Nov 13, 2023 at 3:35 AM Hang Ruan wrote: > Hi, Razin. > > It seems like the issue you shared is a differe

Re: Apache Flink + Java 17 error module

Hi Patricia, Try to use this: --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED Regards, Xiangyu patricia lee 于2023年11月14日周二 15:43写道： > Hi, > > > I upgraded the project to Flink 1.18.0 and Java 17. I am also using > flink-kafka-connector 3.0.1-1.18 from

Re: dependency error with latest Kafka connector

Isn't it expected that it points to 1.17? That version of the Kafka connector is meant to be compatible with both Flink 1.17 and 1.18, right? So the older version should be specified so that the consumer can decide which Flink version to compile against, otherwise the build tool could silently upda

Re: dependency error with latest Kafka connector

Hi Günterh, It looks like a problem with the Kafka connector release. https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.0.1-1.18 Compile dependencies are still pointing to Flink 1.17. Release person is already contacted about this or will be contacted soon. Best regards

Re: Handling default fields in Avro messages using Flink SQL

Hi, Dale. I think there are two choices to try. 1. As the reply in #22427[1], use the SQL function `COALESCE`. 2. Modify the code in Avro format by yourself. There is some work to do for the choice 2. First, you need to pass the default value in Schema, which does not contain the default value no

Re: Offset lost with AT_LEAST_ONCE kafka delivery guarantees

Hey Alexander, Thanks for the feedback and apologies for my late reply. This validates my understanding of AT_LEAST_ONCE wrt the kafka producer. I tried to reproduce the issue, but came back empty handed. As you pointed out the culprit could be a call to an external, non-idempotent, api. I'll f

Re: Kafka Topic Permissions Failure

Hi, Razin. It seems like the issue you shared is a different problem from yours. They have different error messages. Have you ever tried to consume this topic using the Kafka java client[1] by yourself to make sure you could access the topic normally? Best, Hang [1] https://developer.confluent.i

Re: Flink operator autoscaler scaling down

Hi Yang, We're always open to changes / additions to the autoscaler logic and metric collection. Preferably, we change these directly in the autoscaler implementation, without adding additional processes or controllers. Let us know how your experiments go! If you want to contribute, a JIRA with a

Re: Continuous errors with Azure ABFSS

After enabling some more logging for the storage account, I figured out the errors correspond to 404 PathNotFound responses. My guess is the file system checks the status of a path to see if it exists or not before trying to write to it, in this case for _metadata files from each new checkpoint. Se

Re: Flink-1.18 mvn test error

Hi,Yuxin Tan: Thank you very much. My problem has been resolved. Best, Zbz

Re: FLINK CONNECTOR 1.18 and Kafka 3.4.1

Hi, I used the 3.0.1-1.18 connector flink 1.18.0 but it seems the *OffsetResetStrategy* no longer exists. Is there any way we can achieve the same startingoffset reset strategy in flink? E.g. (Kafka connector 1.17.1) kafka.setStartingOffsets(OffsetsInitializer.committedOffsets( *OffsetResetStra

Re: Flink-1.18 mvn test error

Hi, zhao, IIRC, it seems that the error occurred because you ran the test before packaging. While running in flink-clients/pom.xml, maven can not find the packages. Based on your command, it seems that you want to run the tests. Have you tried running "mvn clean package" or "mvn clean install" d

<    4   5   6   7   8   9   10   11   12   13   >