Need help with executing Flink CLI for native Kubernetes deployment

Hi Community, Yang, I am new to Flink on native Kubernetes and I am trying to do a POC for native Kubernetes application mode on Oracle Cloud Infrastructure. I was following the documentation here step by step: [1] I am using Flink 1.12.1, Scala 2.11, java 11. I was able to create a native Kube

Re: How to visualize the results of Flink processing or aggregation?

Prometheus is a metrics system; you can use Flink's Prometheus metrics reporter to send metrics to Prometheus. Grafana can also be connected to influxdb, and to databases like mysql and postgresql, for which sinks are available. And the Elasticsearch sink can be used to create visualizations with

Restore from Checkpoint from local Standalone Job

Hello I was reading this: https://stackoverflow.com/questions/61010970/flink-resume-from-externalised-checkpoint-question I am trying to run a standalone job on my local with a single job manager and task manager. I have enabled checkpointing as below: env.setStateBackend(new RocksDBState

Re: [External] : How to visualize the results of Flink processing or aggregation?

Hello Xiong, You can expose monitors through Metric system of Flink. https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html Metrics can be exposed by metric reporter: https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/metric_reporters.html That includes Promet

How to visualize the results of Flink processing or aggregation?

Hi All, I am new to Flink, so forgive me if it is a naive question. The context is: We have a data streaming coming in, and we will use Flink applications to do the processing or aggregations. After the processing or aggregation, we need some approaches to visualize the results, to either build a

Re: [BULK]Re: [SURVEY] Remove Mesos support

+1 for officially deprecating this component for the 1.13 release. Cheers, Till On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf wrote: > Hi Matthias, > > Thank you for following up on this. +1 to officially deprecate Mesos in > the code and documentation, too. It will be confusing for users if

Re: State size increasing exponentially in Flink v1.9

Hi, If using RocksDB state backend, why it would occur `CopyOnWriteStateMap`? CopyOnWriteStateMap should only exist in heap based state-backend. Best Yun Tang From: Chesnay Schepler Sent: Friday, March 26, 2021 18:45 To: Almeida, Julius ; user@flink.apache.org

Re: Question about checkpoints and savepoints

Here’s a snippet from the logs, there are no errors in the logs 2021-03-23 13:11:52,247 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - 2021-03-23 13:11:52,249 INFO org.apache.flink.runtime.entr

Re: Could not find a suitable table factory for 'org.apache.flink.table.factories.CatalogFactory' in the classpath

Please try copying the connector jar into the lib/ or opt/ directory. On 3/26/2021 11:59 AM, Yik San Chan wrote: This question is cross-posted on Stack Overflow https://stackoverflow.com/questions/66815572/could-not-find-a-suitable-table-factory-for-org-apache-flink-table-factories-ca

Could not find a suitable table factory for 'org.apache.flink.table.factories.CatalogFactory' in the classpath

This question is cross-posted on Stack Overflow https://stackoverflow.com/questions/66815572/could-not-find-a-suitable-table-factory-for-org-apache-flink-table-factories-ca . I am running a PyFlink program that reads from Hive `mysource` table, does some processing, then writes to Hive `mysink` ta

Re: Fail to cancel perJob for that deregisterApplication is not called

Where exactly did you add your own log message? WebMonitorEndpoint.closeAsync() already logs on it's own whether the shutdown future was completed; meaning that it shouldn't have been necessary to add a separate log message. If you now only see the one you added, chances are that it was added a

Re: State size increasing exponentially in Flink v1.9

Could you show us how you interact with the map state (ideally the full code of your function that accesses the state)? On 3/25/2021 1:13 AM, Almeida, Julius wrote: Hey, Hope you all are doing well! I am using flink v1.9 with RocksDBStateBackend, but over time the state size is increasing e

Re: Flink SQL client: SELECT 'hello world' throws [ERROR] Could not execute SQL statement

Thank you for the information. On Fri, Mar 26, 2021 at 5:14 PM David Anderson wrote: > There needs to be a Flink session cluster available to the SQL client on > which it can run the jobs created by your queries. See the Getting Started > [1] section of the SQL Client documentation for more info

Re: Hadoop is not in the classpath/dependencies

Hey Matthias, Maybe the classpath contains hadoop libraries, but not the HDFS libraries? The "DistributedFileSystem" class needs to be accessible to the classloader. Can you check if that class is available? Best, Robert On Thu, Mar 25, 2021 at 11:10 AM Matthias Seiler < matthias.sei...@campus.t

Re: Flink SQL client: SELECT 'hello world' throws [ERROR] Could not execute SQL statement

There needs to be a Flink session cluster available to the SQL client on which it can run the jobs created by your queries. See the Getting Started [1] section of the SQL Client documentation for more information: The SQL Client is bundled in the regular Flink distribution and thus runnable out-of

Re: reading from jdbc connection

Hey Arran, It seems that the preferred way, even in the Java API is to use a DDL statement: https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/table/api/bridge/java/StreamTableEnvironment.java#L602-L639 Hope this helps! Best, Rober

Flink SQL client: SELECT 'hello world' throws [ERROR] Could not execute SQL statement

The question is cross-posted in Stack Overflow https://stackoverflow.com/questions/66813644/flink-sql-client-select-hello-world-throws-error-could-not-execute-sql-stat . I am following Flink SQL client docs https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sqlClient.html#dependencie

Re: FlinkKafkaConsumer - Broadcast - Initial Load

Hey Sandeep, (Maybe this thread is also relevant: https://lists.apache.org/thread.html/7d56267d4c2344ccb5a774896682d0a3efb38c1c215ef3500c3569a2%40%3Cuser.flink.apache.org%3E ) > My question is how do I initialise the pipeline for the first set of records in the database? i.e. those that are not C

Re: Question about checkpoints and savepoints

Hi, has the "state.savepoints.dir" configuration key the same value as "state.checkpoints.dir"? If not, can you post your configuration keys, and the invocation how you trigger a savepoint? Have you checked the logs? Maybe there's an error message? On Thu, Mar 25, 2021 at 7:17 PM Robert Cullen w

Re: Flink on Minikube

Hey Sandeep, here's a project I've recently worked on, that deploys Flink on Minikube: https://github.com/rmetzger/flink-reactive-mode-k8s-demo The project is pretty big, but I guess you can pick the bits related to the Flink deployment on minikube. On Thu, Mar 25, 2021 at 7:48 PM Sandeep khanzod

Re: Flink job repeated restart failure

Hi Vinaya, java.io.tmpdir is already the fallback and I'm not aware of another level of fallback. Ensuring java.io.tmpdir is valid is also relevant for some third-party libraries that rely on it (e.g. FileSystem that cache local files). It's good practice to set that appropriately. On Fri, Mar 2

Re: pipeline.auto-watermark-interval vs setAutoWatermarkInterval

Thanks for double-checking Dawid and thanks for clarifying, Jark. I will leave the Jira issue open as Jark suggested improving the documentation in that sense. Best, Matthias On Fri, Mar 26, 2021 at 7:43 AM Jark Wu wrote: > IIUC, pipeline.auto-watermak-interval = 0 just disable **periodic** > w