Re: PyFlink MapState with Types.ROW() throws exception

2023-10-05 Thread Elkhan Dadashov
() .build() ) import os os.environ["PYFLINK_GATEWAY_DISABLED"] = "0" On Wed, Oct 4, 2023 at 1:48 PM Elkhan Dadashov wrote: > Hi Flinkers, > > I'm trying to use MapState, where the value will be a list of 'pyflink.common.types.Row'> ty

PyFlink MapState with Types.ROW() throws exception

2023-10-04 Thread Elkhan Dadashov
Hi Flinkers, I'm trying to use MapState, where the value will be a list of type elements. Wanted to check if anyone else faced the same issue while trying to use MapState in PyFlink with complex types. Here is the code: from pyflink.common import Time from pyflink.common.typeinfo import Types

[jira] [Created] (FLINK-33188) PyFlink MapState with Types.ROW() throws exception

2023-10-04 Thread Elkhan Dadashov (Jira)
Elkhan Dadashov created FLINK-33188: --- Summary: PyFlink MapState with Types.ROW() throws exception Key: FLINK-33188 URL: https://issues.apache.org/jira/browse/FLINK-33188 Project: Flink

[jira] [Created] (FLINK-32885) Refactoring: Moving UrlPrefixDecorator into flink-clients so it can be used by RestClusterClient for PyFlink remote execution

2023-08-16 Thread Elkhan Dadashov (Jira)
Elkhan Dadashov created FLINK-32885: --- Summary: Refactoring: Moving UrlPrefixDecorator into flink-clients so it can be used by RestClusterClient for PyFlink remote execution Key: FLINK-32885 URL: https

[jira] [Created] (FLINK-32884) PyFlink remote execution should support URLs with paths and https scheme

2023-08-16 Thread Elkhan Dadashov (Jira)
Elkhan Dadashov created FLINK-32884: --- Summary: PyFlink remote execution should support URLs with paths and https scheme Key: FLINK-32884 URL: https://issues.apache.org/jira/browse/FLINK-32884

[jira] [Created] (FLINK-32728) Metrics are not reported in Python UDF (used inside FlinkSQL) when exception is raised

2023-08-01 Thread Elkhan Dadashov (Jira)
Elkhan Dadashov created FLINK-32728: --- Summary: Metrics are not reported in Python UDF (used inside FlinkSQL) when exception is raised Key: FLINK-32728 URL: https://issues.apache.org/jira/browse/FLINK-32728

Metrics are not reported in Python UDF (used inside FlinkSQL) when exception is raised

2023-07-27 Thread Elkhan Dadashov
Hi Flinkers, Wanted to check if anyone else has faced this issue before: When Python UDF (which is used inside FlinkSQL) raises an exception, then metrics get lost and not reported. Facing this issue both in Flink 1.16.2 and FLink 1.17.1 (Python 3.9). If an exception is not raised, then metrics

[jira] [Created] (FLINK-32195) Add SQL Gateway custom headers support

2023-05-25 Thread Elkhan Dadashov (Jira)
Elkhan Dadashov created FLINK-32195: --- Summary: Add SQL Gateway custom headers support Key: FLINK-32195 URL: https://issues.apache.org/jira/browse/FLINK-32195 Project: Flink Issue Type: New

[jira] [Created] (FLINK-32126) When program arg contains two single quotes org.apache.flink.configuration.StructuredOptionsSplitter.consumeInQuotes fails

2023-05-18 Thread Elkhan Dadashov (Jira)
Elkhan Dadashov created FLINK-32126: --- Summary: When program arg contains two single quotes org.apache.flink.configuration.StructuredOptionsSplitter.consumeInQuotes fails Key: FLINK-32126 URL: https

Re: Is there a way to control the parallelism of auto-generated Flink operators of the FlinkSQL job graph?

2023-03-24 Thread Elkhan Dadashov
Checking with the community again, if anyone explored this before. Thanks. On Fri, Mar 17, 2023 at 1:56 PM Elkhan Dadashov wrote: > Dear Flink developers, > > Wanted to check, if there is a way to control the parallelism of > auto-generated Flink operators of the FlinkS

Is there a way to control the parallelism of auto-generated Flink operators of the FlinkSQL job graph?

2023-03-17 Thread Elkhan Dadashov
Dear Flink developers, Wanted to check, if there is a way to control the parallelism of auto-generated Flink operators of the FlinkSQL job graph? In Java API, it is possible to have full control of the parallelism of each operator. On FlinkSQL some source and sink connectors support

Want to help upgrading Flink ElasticSearch connector

2022-10-27 Thread Elkhan Dadashov
Hello all, Want to help upgrade the Flink ElasticSearch connector to 8.X.X version. `elasticsearch-7` connector uses ElasticSearch `7.10.2` version. https://github.com/apache/flink-connector-elasticsearch/blob/main/flink-connector-elasticsearch7/pom.xml#L39 When the ElasticSearch server side

[jira] [Created] (FLINK-29785) Upgrade Flink Elasticsearch-7 connector elasticsearch.version to 7.17.0

2022-10-27 Thread Elkhan Dadashov (Jira)
Elkhan Dadashov created FLINK-29785: --- Summary: Upgrade Flink Elasticsearch-7 connector elasticsearch.version to 7.17.0 Key: FLINK-29785 URL: https://issues.apache.org/jira/browse/FLINK-29785

Re: [VOTE] FLIP-150: Introduce Hybrid Source

2021-07-01 Thread Elkhan Dadashov
+1 (non-binding) On 2021/07/01 05:49:44 蒋晓峰 wrote: > Hi everyone, > > > > > Thanks for all the feedback to Hybrid Source so far. Based on the > discussion[1] we seem to have consensus, so I would like to start a vote on > FLIP-150 for which the FLIP has also been updated[2]. > > > > >

ValueState with pure Java class keeping lists/map vs ListState/MapState, which one is a recommended way?

2020-01-17 Thread Elkhan Dadashov
Hi Flinkers, Was curious about if there is any performance(memory/speed) difference between these two options: in window process functions, when keeping state: *1) Create a single ValueState, and store state in pure Java objects* class MyClass { List listOtherClass; Map

Re: How to handle Flink Job with 400MB+ Uberjar with 800+ containers ?

2019-08-30 Thread Elkhan Dadashov
tainers. We'd like to contribute it to the >>> community very soon. >>> >>> 3. We deploys a timeout timer for each launching container. If a task >>> manager does not register in time after its container has been launched, a >>> new container will be alloc

Will StreamingFileSink.forBulkFormat(...) support overriding OnCheckpointRollingPolicy?

2019-07-22 Thread Elkhan Dadashov
Hi Flink Dev team, Will StreamingFileSink.forBulkFormat(...) support overriding OnCheckpointRollingPolicy? Does anyone use StreamingFileSink *with checkpoint disabled *for writing Parquet output files? The output parquet files are generated, but they are empty, and stay in *inprogress* state,