[GitHub] [flink-connector-jdbc] boring-cyborg[bot] commented on pull request #26: [Improve]Increase the valid of judging connection.
boring-cyborg[bot] commented on PR #26: URL: https://github.com/apache/flink-connector-jdbc/pull/26#issuecomment-1463414908 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] X-czh commented on pull request #21673: [FLINK-30513] Cleanup HA storage path on cluster termination
X-czh commented on PR #21673: URL: https://github.com/apache/flink/pull/21673#issuecomment-1463414646 Sorry for that. I'll make sure that each individual commit conforms to the code format standard in the future. Thank you again for your hard work! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31377) BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0
[
https://issues.apache.org/jira/browse/FLINK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698792#comment-17698792
]
Sergey Nuyanzin commented on FLINK-31377:
-
yes i saw the unittest from the description, that's why I was asking namely
end-to-end case...
I worry that without fixing https://issues.apache.org/jira/browse/FLINK-27438
it brings no benefit to the end user
> BinaryArrayData getArray/getMap should Handle null correctly AssertionError:
> valueArraySize (-6) should >= 0
> -
>
> Key: FLINK-31377
> URL: https://issues.apache.org/jira/browse/FLINK-31377
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
>
> you can reproduce this error below. and reason is in ARRAY_CONTAINS
> {code:java}
> if the needle is a Map NOT NULL,and the array has null element.
> this bellowing will cause getElementOrNull(ArrayData array, int pos) only can
> handle not null. so it throw exception
> /*elementGetter =
> ArrayData.createElementGetter(needleDataType.getLogicalType());*/,
> {code}
>
> {code:java}
> // code placeholder
> Stream getTestSetSpecs() {
> return Stream.of(
> TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
> .onFieldsWithData(
> new Map[] {
> null,
> CollectionUtil.map(entry(1, "a"), entry(2,
> "b")),
> CollectionUtil.map(entry(3, "c"), entry(4,
> "d")),
> },
> null)
> .andDataTypes(
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
> DataTypes.STRING())),
> DataTypes.STRING())
> .testResult(
> $("f0").arrayContains(
> CollectionUtil.map(entry(3, "c"),
> entry(4, "d"))),
> "ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
> true,
> DataTypes.BOOLEAN()));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30996) Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka
[ https://issues.apache.org/jira/browse/FLINK-30996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698790#comment-17698790 ] Mason Chen commented on FLINK-30996: Let's save you some time ;). I'll help do that–I have some spare cycles. This will effect the Flink 1.17 rc1 btw > Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka > --- > > Key: FLINK-30996 > URL: https://issues.apache.org/jira/browse/FLINK-30996 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: kafka-4.0.0 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: kafka-4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31393) HsFileDataManager use an incorrect default timeout
[
https://issues.apache.org/jira/browse/FLINK-31393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-31393:
---
Priority: Critical (was: Blocker)
> HsFileDataManager use an incorrect default timeout
> --
>
> Key: FLINK-31393
> URL: https://issues.apache.org/jira/browse/FLINK-31393
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Critical
>
> For batch shuffle(i.e. hybrid shuffle & sort-merge shuffle), If there is a
> fierce contention of the batch shuffle read memory, it will throw a
> {{TimeoutException}} to fail downstream task to release memory. But for
> hybrid shuffle, It uses an incorrect default timeout(5ms), this will make the
> job very easy to fail.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31305) KafkaWriter doesn't wait for errors for in-flight records before completing flush
[ https://issues.apache.org/jira/browse/FLINK-31305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698789#comment-17698789 ] Mason Chen commented on FLINK-31305: Opened a PR! cc: [~stevenz3wu] [~martijnvisser] [~tzulitai] > KafkaWriter doesn't wait for errors for in-flight records before completing > flush > - > > Key: FLINK-31305 > URL: https://issues.apache.org/jira/browse/FLINK-31305 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.17.0, 1.16.1 >Reporter: Mason Chen >Assignee: Mason Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > The KafkaWriter flushing needs to wait for all in-flight records to send > successfully. This can be achieved by tracking requests and returning a > response from the registered callback from the producer#send() logic. > There is potential for data loss since the checkpoint does not accurately > reflect that all records have been sent successfully, to preserve at least > once semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31305) KafkaWriter doesn't wait for errors for in-flight records before completing flush
[ https://issues.apache.org/jira/browse/FLINK-31305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31305: --- Labels: pull-request-available (was: ) > KafkaWriter doesn't wait for errors for in-flight records before completing > flush > - > > Key: FLINK-31305 > URL: https://issues.apache.org/jira/browse/FLINK-31305 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.17.0, 1.16.1 >Reporter: Mason Chen >Assignee: Mason Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > The KafkaWriter flushing needs to wait for all in-flight records to send > successfully. This can be achieved by tracking requests and returning a > response from the registered callback from the producer#send() logic. > There is potential for data loss since the checkpoint does not accurately > reflect that all records have been sent successfully, to preserve at least > once semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-kafka] mas-chen opened a new pull request, #6: [FLINK-31305] Propagate producer exceptions outside of mailbox execut…
mas-chen opened a new pull request, #6: URL: https://github.com/apache/flink-connector-kafka/pull/6 …or so that checkpoints can correctly fail --> ## What is the purpose of the change Fixes a regression when the checkpoint completes although there is an error. The exception is never thrown before checkpoint completing because the exception throwing logic is enqueued to separate task, during checkpointing. ## Brief change log - Propagate exception and throw them in write/flush/close methods. ## Verifying this change This change added tests and can be verified as follows: - Added unit tests and ran existing unit tests, integration tests, and e2e tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): yes - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22146: [FLINK-31157][release] Add release note for version 1.17
snuyanzin commented on code in PR #22146: URL: https://github.com/apache/flink/pull/22146#discussion_r1132040358 ## docs/content.zh/release-notes/flink-1.17.md: ## @@ -0,0 +1,201 @@ +--- +title: "Release Notes - Flink 1.17" +--- + + +# Release notes - Flink 1.17 + +These release notes discuss important aspects, such as configuration, behavior, or dependencies, +that changed between Flink 1.16 and Flink 1.17. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.17. + + +### Clusters & Deployment + + Only one Zookeeper version is bundled in `flink-dist` +# [FLINK-30237](https://issues.apache.org/jira/browse/FLINK-30237) +The Flink distribution no longer bundles 2 different Zookeeper client jars (one in lib/opt +respectively). Instead, only 1 client will be bundled within the flink-dist jar. This has no +effect on the supported Zookeeper server versions. + + +### Table API & SQL + + Incompatible plan change of event time temporal join on an upsert source +# [FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849) +A correctness issue when do event time temporal join with a versioned table backed by an upsert +source was resolved. When the right input of the join is an upsert source, it no longer generates +a ChangelogNormalize node for it. This is an incompatible plan change compare to 1.16.0. + + Incompatible plan change of filter after temporal join +# [FLINK-28988](https://issues.apache.org/jira/browse/FLINK-28988) +After the patch applied, the filter will not be pushed down into both inputs of the event time +temporal join. Note this may cause incompatible plan changes compare to 1.16.0, e.g., when left +input is an upsert source(use upsert-kafka connector), the query plan will remove the +ChangelogNormalize node from which appeared in 1.16.0. + +### Connectors & Libraries + + Remove cassandra connector from master branch +# [FLINK-30312](https://issues.apache.org/jira/browse/FLINK-30312) +The Cassandra connector has been externalized and is no longer released as part of the main Flink +release. Downloads can be found at https://flink.apache.org/downloads.html and the +source code at https://github.com/apache/flink-connector-cassandra. + + Remove Pulsar connector from master branch +# [FLINK-30397](https://issues.apache.org/jira/browse/FLINK-30397) +The Pulsar connector has been externalized and is no longer bundled and released as part of the +main Flink release. Downloads can be found at https://flink.apache.org/downloads.html and the +source code at https://github.com/apache/flink-connector-pulsar. + + Remove HCatalog +# [FLINK-29669](https://issues.apache.org/jira/browse/FLINK-29669) +The HCatalog connector has been removed from Flink. You can use the Hive connector as a replacement. + + Remove Gelly +# [FLINK-29668](https://issues.apache.org/jira/browse/FLINK-29668) +Gelly has been removed from Flink. Current users of Gelly should not upgrade to Flink 1.17 but +stay on an older version. If you're looking for iterations support, you could investigate +[Flink ML Iteration](https://nightlies.apache.org/flink/flink-ml-docs-stable/docs/development/iteration/) +as a potential successor. + + Support watermark alignment of source splits +# [FLINK-28853](https://issues.apache.org/jira/browse/FLINK-28853) +Since Flink 1.17, source connectors have to implement watermark alignment of source split in order +to use the watermark alignment feature. The required methods to implement are: +`SourceReader#pauseOrResumeSplits` and `SplitReader#pauseOrResumeSplits`. + +If you are migrating from +Flink <= 1.16.x, and you were using watermark alignment, but at the same time you are not able to +upgrade/modify your connector, you can disable per split alignment via setting +`pipeline.watermark-alignment.allow-unaligned-source-splits` to true. Note that by doing so, +watermark alignment will be working properly only when your number of splits equals to the +parallelism of the source operator. + + Remove deprecated MiniClusterResource +# [FLINK-29548](https://issues.apache.org/jira/browse/FLINK-29548) +The deprecated `MiniClusterResource` in `flink-test-utils` has been removed. The +`MiniClusterWithClientResource` is a drop-in replacement. + + Kinesis connector doesn't shade jackson dependency +# [FLINK-14896](https://issues.apache.org/jira/browse/FLINK-14896) +Shade and relocate transitive Jackson dependencies of `flink-connector-kinesis`. If your Flink job +was transitively relying on the these, you may need to include additional Jackson dependencies into +your project. + +### Runtime & Coordination + + Speculative execution take input data amount into account when detecting slow tasks +# [FLINK-30707](https://issues.apache.org/jira/browse/FLINK-30707) +The slow task detecting is improved for speculative execution.
[GitHub] [flink] snuyanzin commented on a diff in pull request #22146: [FLINK-31157][release] Add release note for version 1.17
snuyanzin commented on code in PR #22146: URL: https://github.com/apache/flink/pull/22146#discussion_r1132039260 ## docs/content/release-notes/flink-1.17.md: ## @@ -0,0 +1,201 @@ +--- +title: "Release Notes - Flink 1.17" +--- + + +# Release notes - Flink 1.17 + +These release notes discuss important aspects, such as configuration, behavior, or dependencies, +that changed between Flink 1.16 and Flink 1.17. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.17. + + +### Clusters & Deployment + + Only one Zookeeper version is bundled in `flink-dist` +# [FLINK-30237](https://issues.apache.org/jira/browse/FLINK-30237) +The Flink distribution no longer bundles 2 different Zookeeper client jars (one in lib/opt +respectively). Instead, only 1 client will be bundled within the flink-dist jar. This has no +effect on the supported Zookeeper server versions. + + +### Table API & SQL + + Incompatible plan change of event time temporal join on an upsert source +# [FLINK-29849](https://issues.apache.org/jira/browse/FLINK-29849) +A correctness issue when do event time temporal join with a versioned table backed by an upsert +source was resolved. When the right input of the join is an upsert source, it no longer generates +a ChangelogNormalize node for it. This is an incompatible plan change compare to 1.16.0. + + Incompatible plan change of filter after temporal join +# [FLINK-28988](https://issues.apache.org/jira/browse/FLINK-28988) +After the patch applied, the filter will not be pushed down into both inputs of the event time +temporal join. Note this may cause incompatible plan changes compare to 1.16.0, e.g., when left +input is an upsert source(use upsert-kafka connector), the query plan will remove the +ChangelogNormalize node from which appeared in 1.16.0. + +### Connectors & Libraries + + Remove cassandra connector from master branch +# [FLINK-30312](https://issues.apache.org/jira/browse/FLINK-30312) +The Cassandra connector has been externalized and is no longer released as part of the main Flink +release. Downloads can be found at https://flink.apache.org/downloads.html and the +source code at https://github.com/apache/flink-connector-cassandra. + + Remove Pulsar connector from master branch +# [FLINK-30397](https://issues.apache.org/jira/browse/FLINK-30397) +The Pulsar connector has been externalized and is no longer bundled and released as part of the +main Flink release. Downloads can be found at https://flink.apache.org/downloads.html and the +source code at https://github.com/apache/flink-connector-pulsar. + + Remove HCatalog +# [FLINK-29669](https://issues.apache.org/jira/browse/FLINK-29669) +The HCatalog connector has been removed from Flink. You can use the Hive connector as a replacement. + + Remove Gelly +# [FLINK-29668](https://issues.apache.org/jira/browse/FLINK-29668) +Gelly has been removed from Flink. Current users of Gelly should not upgrade to Flink 1.17 but +stay on an older version. If you're looking for iterations support, you could investigate +[Flink ML Iteration](https://nightlies.apache.org/flink/flink-ml-docs-stable/docs/development/iteration/) +as a potential successor. + + Support watermark alignment of source splits +# [FLINK-28853](https://issues.apache.org/jira/browse/FLINK-28853) +Since Flink 1.17, source connectors have to implement watermark alignment of source split in order +to use the watermark alignment feature. The required methods to implement are: +`SourceReader#pauseOrResumeSplits` and `SplitReader#pauseOrResumeSplits`. + +If you are migrating from +Flink <= 1.16.x, and you were using watermark alignment, but at the same time you are not able to +upgrade/modify your connector, you can disable per split alignment via setting +`pipeline.watermark-alignment.allow-unaligned-source-splits` to true. Note that by doing so, +watermark alignment will be working properly only when your number of splits equals to the +parallelism of the source operator. + + Remove deprecated MiniClusterResource +# [FLINK-29548](https://issues.apache.org/jira/browse/FLINK-29548) +The deprecated `MiniClusterResource` in `flink-test-utils` has been removed. The +`MiniClusterWithClientResource` is a drop-in replacement. + + Kinesis connector doesn't shade jackson dependency +# [FLINK-14896](https://issues.apache.org/jira/browse/FLINK-14896) +Shade and relocate transitive Jackson dependencies of `flink-connector-kinesis`. If your Flink job +was transitively relying on the these, you may need to include additional Jackson dependencies into +your project. + +### Runtime & Coordination + + Speculative execution take input data amount into account when detecting slow tasks +# [FLINK-30707](https://issues.apache.org/jira/browse/FLINK-30707) +The slow task detecting is improved for speculative execution.
[jira] [Commented] (FLINK-26603) [Umbrella] Decouple Hive with Flink planner
[ https://issues.apache.org/jira/browse/FLINK-26603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698786#comment-17698786 ] Martijn Visser commented on FLINK-26603: [~luoyuxia] Given that this wasn't possible to achieve in 1.16 or 1.17, do you think there is a realistic and committable timeline for completing this ticket? > [Umbrella] Decouple Hive with Flink planner > --- > > Key: FLINK-26603 > URL: https://issues.apache.org/jira/browse/FLINK-26603 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / Planner >Reporter: luoyuxia >Priority: Major > Fix For: 1.17.0 > > > To support Hive dialect with Flink, we have implemented FLIP-123, FLIP-152. > But it also brings much maintenance burden and complexity for it mixes some > logic specific to Hive with Flink planner. We should remove such logic from > Flink planner and make it totally decouple with Flink planner. > With this ticket, we expect: > 1: there won't be any specific logic to Hive in planner module > 2: remove flink-sql-parser-hive from flink-table module > 3: remove the planner dependency in flink-connector-hive > I'll update more details after investigation. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-kafka] MartijnVisser closed pull request #2: [FLINK-30880] Update flink version to 1.16.1
MartijnVisser closed pull request #2: [FLINK-30880] Update flink version to 1.16.1 URL: https://github.com/apache/flink-connector-kafka/pull/2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] MartijnVisser commented on pull request #2: [FLINK-30880] Update flink version to 1.16.1
MartijnVisser commented on PR #2: URL: https://github.com/apache/flink-connector-kafka/pull/2#issuecomment-1463395015 Superseded by https://github.com/apache/flink-connector-kafka/pull/4/commits/0c1b2e5309c29425437dbad557bdcf2643a28039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-30052) Move existing Kafka connector code from Flink repo to dedicated Kafka repo
[ https://issues.apache.org/jira/browse/FLINK-30052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-30052. -- Fix Version/s: kafka-3.0.0 Resolution: Fixed Fixed via https://github.com/apache/flink-connector-kafka/pull/1 > Move existing Kafka connector code from Flink repo to dedicated Kafka repo > -- > > Key: FLINK-30052 > URL: https://issues.apache.org/jira/browse/FLINK-30052 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Martijn Visser >Assignee: Mason Chen >Priority: Major > Labels: pull-request-available > Fix For: kafka-3.0.0 > > > Instructions guide can be found at > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-hbase] MartijnVisser commented on pull request #5: [FLINK-30349] Sync missing HBase e2e tests to external repo
MartijnVisser commented on PR #5: URL: https://github.com/apache/flink-connector-hbase/pull/5#issuecomment-1463393988 @ferenc-csaky With @zentol his comments, do you think you could continue working on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-29825) Improve benchmark stability
[ https://issues.apache.org/jira/browse/FLINK-29825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved FLINK-29825. -- Resolution: Fixed > Improve benchmark stability > --- > > Key: FLINK-29825 > URL: https://issues.apache.org/jira/browse/FLINK-29825 > Project: Flink > Issue Type: Improvement > Components: Benchmarks >Affects Versions: 1.17.0 >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Minor > Labels: pull-request-available > > Currently, regressions are detected by a simple script which may have false > positives and false negatives, especially for benchmarks with small absolute > values, small value changes would cause large percentage changes. see > [here|https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136] > for details. > And all benchmarks are executed on one physical machine, it might happen that > hardware issues affect performance, like "[FLINK-18614] Performance > regression 2020.07.13". > > This ticket aims to improve the precision and recall of the regression-check > script. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] reswqa commented on pull request #21673: [FLINK-30513] Cleanup HA storage path on cluster termination
reswqa commented on PR #21673: URL: https://github.com/apache/flink/pull/21673#issuecomment-1463394127 @XComp Very sorry for this. This PR has two commits, the first is a `hotfix` that only migrates test to junit5. I take it for granted that it will not have code format problems, so it is not reverted together. I will pay attention to this in the future and run a round of CI before doing this. Thank you again @snuyanzin! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29825) Improve benchmark stability
[ https://issues.apache.org/jira/browse/FLINK-29825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698784#comment-17698784 ] Dong Lin commented on FLINK-29825: -- Merged to apache/flink-benchmarks master branch 7d2013a9f401366bc9073857175f434882867bfe > Improve benchmark stability > --- > > Key: FLINK-29825 > URL: https://issues.apache.org/jira/browse/FLINK-29825 > Project: Flink > Issue Type: Improvement > Components: Benchmarks >Affects Versions: 1.17.0 >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Minor > Labels: pull-request-available > > Currently, regressions are detected by a simple script which may have false > positives and false negatives, especially for benchmarks with small absolute > values, small value changes would cause large percentage changes. see > [here|https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136] > for details. > And all benchmarks are executed on one physical machine, it might happen that > hardware issues affect performance, like "[FLINK-18614] Performance > regression 2020.07.13". > > This ticket aims to improve the precision and recall of the regression-check > script. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30996) Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka
[ https://issues.apache.org/jira/browse/FLINK-30996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-30996. -- Fix Version/s: kafka-4.0.0 Resolution: Fixed Fixed via https://github.com/apache/flink-connector-kafka/pull/4 > Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka > --- > > Key: FLINK-30996 > URL: https://issues.apache.org/jira/browse/FLINK-30996 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: kafka-4.0.0 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: kafka-4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30996) Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka
[ https://issues.apache.org/jira/browse/FLINK-30996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698782#comment-17698782 ] Martijn Visser commented on FLINK-30996: [~mason6345] Oh duh, I forgot to close this. Yes, let's work on FLINK-30859. Do you want to do it and I review, or do you want me to do it and you review? > Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka > --- > > Key: FLINK-30996 > URL: https://issues.apache.org/jira/browse/FLINK-30996 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: kafka-4.0.0 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] jiangxin369 commented on pull request #218: [FLINK-31306] Add Servable for PipelineModel
jiangxin369 commented on PR #218: URL: https://github.com/apache/flink-ml/pull/218#issuecomment-1463393272 @lindong28 Thanks for the review, I've updated the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-benchmarks] lindong28 merged pull request #66: [FLINK-29825] Replace median-based regression detect algorithm with max/min-based
lindong28 merged PR #66: URL: https://github.com/apache/flink-benchmarks/pull/66 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22146: [FLINK-31157][release] Add release note for version 1.17
flinkbot commented on PR #22146: URL: https://github.com/apache/flink/pull/22146#issuecomment-1463391299 ## CI report: * 5eb387243eb4e0d2cbdf73556444f38c9dd365fe UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-30640) Unstable test in CliClientITCase
[ https://issues.apache.org/jira/browse/FLINK-30640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698778#comment-17698778 ] Matthias Pohl commented on FLINK-30640: --- I'm increasing the priority of this issue to Blocker until it's clear whether it's a only test code issue or more severe [~fsk119] [~tartarus] [~lsy] may you have a look at it? > Unstable test in CliClientITCase > > > Key: FLINK-30640 > URL: https://issues.apache.org/jira/browse/FLINK-30640 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client >Affects Versions: 1.17.0 >Reporter: yuzelin >Assignee: dalongliu >Priority: Blocker > Labels: pull-request-available, test-stability > Fix For: 1.17.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44743=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4] > > The failed test can work normally in my local environment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30996) Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka
[ https://issues.apache.org/jira/browse/FLINK-30996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698779#comment-17698779 ] Mason Chen commented on FLINK-30996: [~martijnvisser] Looks like this is done. Should we work on https://issues.apache.org/jira/browse/FLINK-30859 to remove from master/release-1.17? I can help if needed > Sync Kafka 1.17 commits from apache/flink repo to flink-connector-kafka > --- > > Key: FLINK-30996 > URL: https://issues.apache.org/jira/browse/FLINK-30996 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: kafka-4.0.0 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31377) BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0
[
https://issues.apache.org/jira/browse/FLINK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698780#comment-17698780
]
jackylau commented on FLINK-31377:
--
this https://issues.apache.org/jira/browse/FLINK-27438 just a pure sql level,
which is not blocked.
> BinaryArrayData getArray/getMap should Handle null correctly AssertionError:
> valueArraySize (-6) should >= 0
> -
>
> Key: FLINK-31377
> URL: https://issues.apache.org/jira/browse/FLINK-31377
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
>
> you can reproduce this error below. and reason is in ARRAY_CONTAINS
> {code:java}
> if the needle is a Map NOT NULL,and the array has null element.
> this bellowing will cause getElementOrNull(ArrayData array, int pos) only can
> handle not null. so it throw exception
> /*elementGetter =
> ArrayData.createElementGetter(needleDataType.getLogicalType());*/,
> {code}
>
> {code:java}
> // code placeholder
> Stream getTestSetSpecs() {
> return Stream.of(
> TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
> .onFieldsWithData(
> new Map[] {
> null,
> CollectionUtil.map(entry(1, "a"), entry(2,
> "b")),
> CollectionUtil.map(entry(3, "c"), entry(4,
> "d")),
> },
> null)
> .andDataTypes(
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
> DataTypes.STRING())),
> DataTypes.STRING())
> .testResult(
> $("f0").arrayContains(
> CollectionUtil.map(entry(3, "c"),
> entry(4, "d"))),
> "ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
> true,
> DataTypes.BOOLEAN()));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31377) BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0
[
https://issues.apache.org/jira/browse/FLINK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698777#comment-17698777
]
jackylau commented on FLINK-31377:
--
[~Sergey Nuyanzin] this unit test code can reproduce it
{code:java}
// code placeholder
Stream getTestSetSpecs() {
return Stream.of(
TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
.onFieldsWithData(
new Map[] {
null,
CollectionUtil.map(entry(1, "a"), entry(2,
"b")),
CollectionUtil.map(entry(3, "c"), entry(4,
"d")),
},
null)
.andDataTypes(
DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
DataTypes.STRING())),
DataTypes.STRING())
.testResult(
$("f0").arrayContains(
CollectionUtil.map(entry(3, "c"),
entry(4, "d"))),
"ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
true,
DataTypes.BOOLEAN()));
} {code}
> BinaryArrayData getArray/getMap should Handle null correctly AssertionError:
> valueArraySize (-6) should >= 0
> -
>
> Key: FLINK-31377
> URL: https://issues.apache.org/jira/browse/FLINK-31377
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
>
> you can reproduce this error below. and reason is in ARRAY_CONTAINS
> {code:java}
> if the needle is a Map NOT NULL,and the array has null element.
> this bellowing will cause getElementOrNull(ArrayData array, int pos) only can
> handle not null. so it throw exception
> /*elementGetter =
> ArrayData.createElementGetter(needleDataType.getLogicalType());*/,
> {code}
>
> {code:java}
> // code placeholder
> Stream getTestSetSpecs() {
> return Stream.of(
> TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
> .onFieldsWithData(
> new Map[] {
> null,
> CollectionUtil.map(entry(1, "a"), entry(2,
> "b")),
> CollectionUtil.map(entry(3, "c"), entry(4,
> "d")),
> },
> null)
> .andDataTypes(
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
> DataTypes.STRING())),
> DataTypes.STRING())
> .testResult(
> $("f0").arrayContains(
> CollectionUtil.map(entry(3, "c"),
> entry(4, "d"))),
> "ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
> true,
> DataTypes.BOOLEAN()));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31157) Propose a pull request for website updates
[
https://issues.apache.org/jira/browse/FLINK-31157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31157:
---
Labels: pull-request-available (was: )
> Propose a pull request for website updates
> --
>
> Key: FLINK-31157
> URL: https://issues.apache.org/jira/browse/FLINK-31157
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> The final step of building the candidate is to propose a website pull request
> containing the following changes:
> # update
> [apache/flink-web:_config.yml|https://github.com/apache/flink-web/blob/asf-site/_config.yml]
> ## update {{FLINK_VERSION_STABLE}} and {{FLINK_VERSION_STABLE_SHORT}} as
> required
> ## update version references in quickstarts ({{{}q/{}}} directory) as
> required
> ## (major only) add a new entry to {{flink_releases}} for the release
> binaries and sources
> ## (minor only) update the entry for the previous release in the series in
> {{flink_releases}}
> ### Please pay notice to the ids assigned to the download entries. They
> should be unique and reflect their corresponding version number.
> ## add a new entry to {{release_archive.flink}}
> # add a blog post announcing the release in _posts
> # add a organized release notes page under docs/content/release-notes and
> docs/content.zh/release-notes (like
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/release-notes/flink-1.15/]).
> The page is based on the non-empty release notes collected from the issues,
> and only the issues that affect existing users should be included (e.g.,
> instead of new functionality). It should be in a separate PR since it would
> be merged to the flink project.
> (!) Don’t merge the PRs before finalizing the release.
>
>
> h3. Expectations
> * Website pull request proposed to list the
> [release|http://flink.apache.org/downloads.html]
> * (major only) Check {{docs/config.toml}} to ensure that
> ** the version constants refer to the new version
> ** the {{baseurl}} does not point to {{flink-docs-master}} but
> {{flink-docs-release-X.Y}} instead
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] PatrickRen opened a new pull request, #22146: [FLINK-31157][release] Add release note for version 1.17
PatrickRen opened a new pull request, #22146: URL: https://github.com/apache/flink/pull/22146 ## What is the purpose of the change This pull request adds release note for version 1.17. ## Brief change log - Add release note for 1.17 ## Verifying this change This change is a doc-only without test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-benchmarks] fredia commented on pull request #66: [FLINK-29825] Replace median-based regression detect algorithm with max/min-based
fredia commented on PR #66: URL: https://github.com/apache/flink-benchmarks/pull/66#issuecomment-1463387085 @lindong28 Thanks for the review and suggestions. > do you have numbers showing how this algorithm performs in production? For example, is there false positive and false negative rate of this algorithm? I have selected several benchmarks to test the precision and recall of this algorithm. Here are some data: | true positive | false positive | true negative | false negative | precision | recall -- | -- | -- | -- | -- | -- | -- [arrayKeyBy](http://codespeed.dak8s.net:8000/timeline/#/?exe=1,5,8,9=arrayKeyBy=2=1000=off=on=on) | 42 | 2 | 840 | 10 | 0.96 | 0.83 [serializerRow_Java11](http://codespeed.dak8s.net:8000/timeline/#/?exe=5,6,8,9=serializerRow=2=1000=off=on=on) | 4 | 1 | 241 | 3 | 0.8 | 0.57 [serializerTuple_Java11](http://codespeed.dak8s.net:8000/timeline/#/?exe=5,6,8,9=serializerTuple=2=1000=off=on=on) | 4 | 24 | 218 | 3 | 0.14 | 0.57 Overall, It performs differently on different benchmarks, but its precision and recall can basically be greater than 50%. Here is a [doc](https://docs.google.com/document/d/1coI4eJsauBtrlS1Z77bhGf-hNtDEXbzuwacG5ZPCMc8/edit?usp=sharing) comparing it to the current median-based algorithm, which contains their false positive and false negative data on different types of benchmarks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-30640) Unstable test in CliClientITCase
[ https://issues.apache.org/jira/browse/FLINK-30640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-30640: -- Priority: Blocker (was: Critical) > Unstable test in CliClientITCase > > > Key: FLINK-30640 > URL: https://issues.apache.org/jira/browse/FLINK-30640 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client >Affects Versions: 1.17.0 >Reporter: yuzelin >Assignee: dalongliu >Priority: Blocker > Labels: pull-request-available, test-stability > Fix For: 1.17.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44743=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4] > > The failed test can work normally in my local environment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31381) UnsupportedOperationException: Unsupported type when convertTypeToSpec: MAP
[
https://issues.apache.org/jira/browse/FLINK-31381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin closed FLINK-31381.
---
Resolution: Duplicate
> UnsupportedOperationException: Unsupported type when convertTypeToSpec: MAP
> ---
>
> Key: FLINK-31381
> URL: https://issues.apache.org/jira/browse/FLINK-31381
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
>
> when i fix this https://issues.apache.org/jira/browse/FLINK-31377, and find
> another bug.
> which is not fixed completely https://github.com/apache/flink/pull/18967/files
> {code:java}
> SELECT array_contains(ARRAY[CAST(null AS MAP), MAP[1, 2]], MAP[1,
> 2]); {code}
> {code:java}
> Caused by: java.lang.UnsupportedOperationException: Unsupported type when
> convertTypeToSpec: MAPat
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1069)
> at
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1091)
> at
> org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.castTo(SqlValidatorUtils.java:82)
> at
> org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.adjustTypeForMultisetConstructor(SqlValidatorUtils.java:74)
> at
> org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.adjustTypeForArrayConstructor(SqlValidatorUtils.java:39)
> at
> org.apache.flink.table.planner.functions.sql.SqlArrayConstructor.inferReturnType(SqlArrayConstructor.java:44)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:504)
> at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:605)at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6218)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6203)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1861)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1852)
> at
> org.apache.flink.table.planner.functions.inference.CallBindingCallContext$1.get(CallBindingCallContext.java:74)
> at
> org.apache.flink.table.planner.functions.inference.CallBindingCallContext$1.get(CallBindingCallContext.java:69)
> at
> org.apache.flink.table.types.inference.strategies.RootArgumentTypeStrategy.inferArgumentType(RootArgumentTypeStrategy.java:58)
> at
> org.apache.flink.table.types.inference.strategies.SequenceInputTypeStrategy.inferInputTypes(SequenceInputTypeStrategy.java:76)
> at
> org.apache.flink.table.planner.functions.inference.TypeInferenceOperandInference.inferOperandTypesOrError(TypeInferenceOperandInference.java:91)
> at org.apache.flink.table. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] zhipeng93 opened a new pull request, #223: [hotfix] Discard watermarks when feed a datastream into iteration body
zhipeng93 opened a new pull request, #223: URL: https://github.com/apache/flink-ml/pull/223 ## What is the purpose of the change This PR proposes to discard watermarks when feed a datastream into iteration body since watermarks are not correctly processed in iteration module. To avoid the possible bugs, we plan to add a java doc to explain that `flink-m-iteration` module cannot deal with watermarks for now. We also leave it as a TODO here and plan to support it in the future. ## Brief change log - Adds a java doc to explain that iterations cannot deal with watermarks correctly. - Removed watermarks for all datastreams that are fed into a iteration body. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (JavaDocs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (FLINK-30640) Unstable test in CliClientITCase
[ https://issues.apache.org/jira/browse/FLINK-30640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl reopened FLINK-30640: --- We have this issue happening again on 1.17: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47010=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=16287 > Unstable test in CliClientITCase > > > Key: FLINK-30640 > URL: https://issues.apache.org/jira/browse/FLINK-30640 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client >Affects Versions: 1.17.0 >Reporter: yuzelin >Assignee: dalongliu >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.17.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44743=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4] > > The failed test can work normally in my local environment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31377) BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0
[
https://issues.apache.org/jira/browse/FLINK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698773#comment-17698773
]
Sergey Nuyanzin commented on FLINK-31377:
-
is there a way to reproduce it with sql or any other end-to-end case?
I'm asking since I suspect it will be blocked by
https://issues.apache.org/jira/browse/FLINK-27438
> BinaryArrayData getArray/getMap should Handle null correctly AssertionError:
> valueArraySize (-6) should >= 0
> -
>
> Key: FLINK-31377
> URL: https://issues.apache.org/jira/browse/FLINK-31377
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
>
> you can reproduce this error below. and reason is in ARRAY_CONTAINS
> {code:java}
> if the needle is a Map NOT NULL,and the array has null element.
> this bellowing will cause getElementOrNull(ArrayData array, int pos) only can
> handle not null. so it throw exception
> /*elementGetter =
> ArrayData.createElementGetter(needleDataType.getLogicalType());*/,
> {code}
>
> {code:java}
> // code placeholder
> Stream getTestSetSpecs() {
> return Stream.of(
> TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
> .onFieldsWithData(
> new Map[] {
> null,
> CollectionUtil.map(entry(1, "a"), entry(2,
> "b")),
> CollectionUtil.map(entry(3, "c"), entry(4,
> "d")),
> },
> null)
> .andDataTypes(
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
> DataTypes.STRING())),
> DataTypes.STRING())
> .testResult(
> $("f0").arrayContains(
> CollectionUtil.map(entry(3, "c"),
> entry(4, "d"))),
> "ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
> true,
> DataTypes.BOOLEAN()));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31210) Azure Pipelines report warning on "no space left on device" in Restore Docker images step
[
https://issues.apache.org/jira/browse/FLINK-31210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694932#comment-17694932
]
Matthias Pohl edited comment on FLINK-31210 at 3/10/23 7:24 AM:
*
[20230301.2|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46642=results]
*
[20230302.3|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46686=results]
*
[20230302.24|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46753=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=841082b6-1a93-5908-4d37-a071f4387a5f]
*
[20230303.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46766=results]
*
[20230304.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46811=results]
*
[20230305.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46811=results]
*
[20230306.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46819=results]
*
[20230306.21|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46869=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=10381]
*
[20230307.2|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46882=results]
* ...
*
[20230310.2|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47009=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=8459]
was (Author: mapohl):
*
[20230301.2|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46642=results]
*
[20230302.3|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46686=results]
*
[20230302.24|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46753=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=841082b6-1a93-5908-4d37-a071f4387a5f]
*
[20230303.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46766=results]
*
[20230304.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46811=results]
*
[20230305.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46811=results]
*
[20230306.03|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46819=results]
*
[20230306.21|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46869=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=10381]
*
[20230307.2|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46882=results]
> Azure Pipelines report warning on "no space left on device" in Restore Docker
> images step
> -
>
> Key: FLINK-31210
> URL: https://issues.apache.org/jira/browse/FLINK-31210
> Project: Flink
> Issue Type: Bug
> Components: Test Infrastructure
>Affects Versions: 1.16.1
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> We're experiencing "no space left on device" issues lately when restoring the
> Docker images. This doesn't make the builds fail, though. It's just reported
> as warning
> {code}
> Loaded image: docker.elastic.co/elasticsearch/elasticsearch:6.8.20
> ApplyLayer exit status 1 stdout: stderr: write
> /usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-City.mmdb: no space
> left on device
> ##[error]Bash exited with code '1'.
> {code}
> All of this happens on Azure machines (not the Alibaba VMs). Therefore, I'm
> not sure whether there's much to do on our side.
> *
> [20230224.3|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46491=logs=f8e16326-dc75-5ba0-3e95-6178dd55bf6c=728e59c6-8078-53a8-7bbe-bb7b0b1f2c63]
> *
> [20230223.18|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46486=results]
> *
> [20230223.6|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46441=results]
> *
> [20230223.2|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46434=results]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31395) AbstractPartitionDiscoverer.discoverPartitions calls remove on immutable collection
Matthias Pohl created FLINK-31395:
-
Summary: AbstractPartitionDiscoverer.discoverPartitions calls
remove on immutable collection
Key: FLINK-31395
URL: https://issues.apache.org/jira/browse/FLINK-31395
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.16.1
Reporter: Matthias Pohl
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47009=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=8459
{{FlinkKafkaConsumerBaseTest.testClosePartitionDiscovererWithCancellation}}
failed because of that.
{code}
[...]
Mar 10 01:48:27 Caused by: java.lang.RuntimeException:
java.lang.UnsupportedOperationException
Mar 10 01:48:27 at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.runWithPartitionDiscovery(FlinkKafkaConsumerBase.java:846)
Mar 10 01:48:27 at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:828)
Mar 10 01:48:27 at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBaseTest.lambda$testNormalConsumerLifecycle$9(FlinkKafkaConsumerBaseTest.java:695)
Mar 10 01:48:27 at
org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:49)
Mar 10 01:48:27 at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
Mar 10 01:48:27 at
java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632)
Mar 10 01:48:27 at
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
Mar 10 01:48:27 at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
Mar 10 01:48:27 at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
Mar 10 01:48:27 at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Mar 10 01:48:27 Caused by: java.lang.UnsupportedOperationException
Mar 10 01:48:27 at java.util.Collections$1.remove(Collections.java:4686)
Mar 10 01:48:27 at
org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.discoverPartitions(AbstractPartitionDiscoverer.java:165)
Mar 10 01:48:27 at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.lambda$createAndStartDiscoveryLoop$2(FlinkKafkaConsumerBase.java:880)
Mar 10 01:48:27 at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31381) UnsupportedOperationException: Unsupported type when convertTypeToSpec: MAP
[
https://issues.apache.org/jira/browse/FLINK-31381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31381:
--
Fix Version/s: (was: 1.18.0)
> UnsupportedOperationException: Unsupported type when convertTypeToSpec: MAP
> ---
>
> Key: FLINK-31381
> URL: https://issues.apache.org/jira/browse/FLINK-31381
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
>
> when i fix this https://issues.apache.org/jira/browse/FLINK-31377, and find
> another bug.
> which is not fixed completely https://github.com/apache/flink/pull/18967/files
> {code:java}
> SELECT array_contains(ARRAY[CAST(null AS MAP), MAP[1, 2]], MAP[1,
> 2]); {code}
> {code:java}
> Caused by: java.lang.UnsupportedOperationException: Unsupported type when
> convertTypeToSpec: MAPat
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1069)
> at
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1091)
> at
> org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.castTo(SqlValidatorUtils.java:82)
> at
> org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.adjustTypeForMultisetConstructor(SqlValidatorUtils.java:74)
> at
> org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.adjustTypeForArrayConstructor(SqlValidatorUtils.java:39)
> at
> org.apache.flink.table.planner.functions.sql.SqlArrayConstructor.inferReturnType(SqlArrayConstructor.java:44)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:504)
> at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:605)at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6218)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6203)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1861)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1852)
> at
> org.apache.flink.table.planner.functions.inference.CallBindingCallContext$1.get(CallBindingCallContext.java:74)
> at
> org.apache.flink.table.planner.functions.inference.CallBindingCallContext$1.get(CallBindingCallContext.java:69)
> at
> org.apache.flink.table.types.inference.strategies.RootArgumentTypeStrategy.inferArgumentType(RootArgumentTypeStrategy.java:58)
> at
> org.apache.flink.table.types.inference.strategies.SequenceInputTypeStrategy.inferInputTypes(SequenceInputTypeStrategy.java:76)
> at
> org.apache.flink.table.planner.functions.inference.TypeInferenceOperandInference.inferOperandTypesOrError(TypeInferenceOperandInference.java:91)
> at org.apache.flink.table. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] XComp commented on pull request #21673: [FLINK-30513] Cleanup HA storage path on cluster termination
XComp commented on PR #21673: URL: https://github.com/apache/flink/pull/21673#issuecomment-1463376405 As a reminder for next time: Please be careful with reverting changes without a commit. The revert broke `master` for some builds (see [20230309.12](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47003=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=3134)). The issue was resolved by [3577edf5](https://github.com/flink-ci/flink-mirror/commit/3577edf57f55177eb5af6ddf1179267e15c69e2f]. Thanks @snuyanzin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-31383) Add support for documenting additionProperties of the REST API payloads.
[
https://issues.apache.org/jira/browse/FLINK-31383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Morávek resolved FLINK-31383.
---
Fix Version/s: 1.18.0
Resolution: Fixed
> Add support for documenting additionProperties of the REST API payloads.
>
>
> Key: FLINK-31383
> URL: https://issues.apache.org/jira/browse/FLINK-31383
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation, Runtime / REST
>Reporter: David Morávek
>Assignee: David Morávek
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> For implementing the request and response body of the resource requirements
> endpoint, we need to be able to document "additionalProperties" because these
> payloads have only top-level dynamic properties of the same type.
>
> An example of what we want to be able to document is:
> {code:java}
> @JsonAnySetter
> @JsonAnyGetter
> @JsonSerialize(keyUsing = JobVertexIDKeySerializer.class)
> @JsonDeserialize(keyUsing = JobVertexIDKeyDeserializer.class)
> private final Map
> jobVertexResourceRequirements;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31383) Add support for documenting additionProperties of the REST API payloads.
[
https://issues.apache.org/jira/browse/FLINK-31383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698767#comment-17698767
]
David Morávek commented on FLINK-31383:
---
master: 84d000cf47b833625fbd5b5f72e48963b3156103
> Add support for documenting additionProperties of the REST API payloads.
>
>
> Key: FLINK-31383
> URL: https://issues.apache.org/jira/browse/FLINK-31383
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation, Runtime / REST
>Reporter: David Morávek
>Assignee: David Morávek
>Priority: Minor
> Labels: pull-request-available
>
> For implementing the request and response body of the resource requirements
> endpoint, we need to be able to document "additionalProperties" because these
> payloads have only top-level dynamic properties of the same type.
>
> An example of what we want to be able to document is:
> {code:java}
> @JsonAnySetter
> @JsonAnyGetter
> @JsonSerialize(keyUsing = JobVertexIDKeySerializer.class)
> @JsonDeserialize(keyUsing = JobVertexIDKeyDeserializer.class)
> private final Map
> jobVertexResourceRequirements;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] dmvk merged pull request #22139: [FLINK-31383] Add support for documenting additionProperties of the R…
dmvk merged PR #22139: URL: https://github.com/apache/flink/pull/22139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31373) PerRoundWrapperOperator should carry epoch information in watermark
[ https://issues.apache.org/jira/browse/FLINK-31373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698766#comment-17698766 ] Zhipeng Zhang commented on FLINK-31373: --- As discussed with [~gaoyunhaii] offline, we agree that the watermark is not correctly processed in iteration module. To avoid the above cases for now, we plan to add a java doc to explain that `flink-m-iteration` module cannot deal with watermarks correctly. We will leave it as a TODO here. > PerRoundWrapperOperator should carry epoch information in watermark > --- > > Key: FLINK-31373 > URL: https://issues.apache.org/jira/browse/FLINK-31373 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Affects Versions: ml-2.2.0 >Reporter: Zhipeng Zhang >Priority: Major > > Currently we use PerRoundWrapperOperator to wrap the normal flink operators > such that they can be used in iterations. > We already contained the epoch information in each record so that we know > which iteration each record belongs to. > However, there is no epoch information when the stream element is a > watermark. This works in most cases, but fail to address the following use > case: > - In DataStreamUtils#withBroadcast, we will cache the elements (including > watermarks) from non-broadcast inputs until the broadcast variables are > ready. When the broadcast variables are ready, once we receive a stream > element we will process the cached elements first. If the received element is > a watermark, the current implementation of iteration module fails > (ProxyOutput#collect throws NPE) since there is no epoch information. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-31394) Fix spark jar name in the create release script for table store
[ https://issues.apache.org/jira/browse/FLINK-31394 ] zhuangchong deleted comment on FLINK-31394: - was (Author: zhuangchong): https://github.com/apache/flink-table-store/pull/591 > Fix spark jar name in the create release script for table store > --- > > Key: FLINK-31394 > URL: https://issues.apache.org/jira/browse/FLINK-31394 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: zhuangchong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-benchmarks] fredia commented on a diff in pull request #66: [FLINK-29825] Replace median-based regression detect algorithm with max/min-based
fredia commented on code in PR #66:
URL: https://github.com/apache/flink-benchmarks/pull/66#discussion_r1132016069
##
regression_report_v2.py:
##
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import urllib
+import urllib2
+
+from regression_report import loadBenchmarkNames
+
+"""
+This is a regression detection algorithm based on the historical
maximum/minimum value, please refer
+to
https://docs.google.com/document/d/1Bvzvq79Ll5yxd1UtC0YzczgFbZPAgPcN3cI0MjVkIag/edit
the detailed design.
+"""
+
+ENVIRONMENT = 2
+
+"""
+Returns a list of benchmark results
+"""
+def loadHistoryData(codespeedUrl, exe, benchmark, baselineSize):
+url = codespeedUrl + 'timeline/json/?' + urllib.urlencode(
+{'exe': exe, 'ben': benchmark, 'env': ENVIRONMENT, 'revs':
baselineSize})
+f = urllib2.urlopen(url)
+response = f.read()
+f.close()
+timelines = json.loads(response)['timelines'][0]
+result = timelines['branches']['master'][exe]
+lessIsbBetter = (timelines['lessisbetter'] == " (less is better)")
+return result, lessIsbBetter
+
+def detectRegression(urlToBenchmark, stds, scores, baselineSize,
minRegressionRatio, minInstabilityMultiplier,
+ direction):
+sustainable_x = [min(scores[i - 3: i]) for i in range(3, baselineSize)]
+baseline_throughput = max(sustainable_x)
+current_throughput = max(scores[-3:])
+current_instability = stds[-1] / current_throughput
+if direction * (1 - current_throughput / baseline_throughput) >
max(minRegressionRatio, minInstabilityMultiplier * current_instability):
+print "<%s|%s> baseline=%s current_value=%s" % (urlToBenchmark,
benchmark, baseline_throughput, current_throughput)
+
+def checkBenchmark(args, exe, benchmark):
+results, lessIsbBetter = loadHistoryData(args.codespeedUrl, exe,
benchmark, args.baselineSize + 3)
+results = list(reversed(results))
+scores = [score for (date, score, deviation, commit, branch) in results]
+stds = [deviation for (date, score, deviation, commit, branch) in results]
+
+urlToBenchmark = args.codespeedUrl + 'timeline/#/?' + urllib.urlencode({
+'ben': benchmark,
+'exe': exe,
+'env': ENVIRONMENT,
+'revs': args.displaySamples,
+'equid': 'off',
+'quarts': 'on',
+'extr': 'on'})
+
+if len(results) < args.baselineSize:
Review Comment:
> Does this mean that a new benchmark will only be included in the
regression detection 30 days after this benchmark is added?
Yes, a new benchmark will only be included in the regression detection 30
days after this benchmark is added.
To detect new benchmark faster, It is a good idea to change the minimum
sample limit to 5.
##
regression_report_v2.py:
##
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import urllib
+import urllib2
+
+from regression_report import loadBenchmarkNames
+
+"""
+This is a
[jira] [Updated] (FLINK-31394) Fix spark jar name in the create release script for table store
[ https://issues.apache.org/jira/browse/FLINK-31394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31394: --- Labels: pull-request-available (was: ) > Fix spark jar name in the create release script for table store > --- > > Key: FLINK-31394 > URL: https://issues.apache.org/jira/browse/FLINK-31394 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: zhuangchong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31394) Fix spark jar name in the create release script for table store
[ https://issues.apache.org/jira/browse/FLINK-31394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698762#comment-17698762 ] zhuangchong commented on FLINK-31394: - https://github.com/apache/flink-table-store/pull/591 > Fix spark jar name in the create release script for table store > --- > > Key: FLINK-31394 > URL: https://issues.apache.org/jira/browse/FLINK-31394 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: zhuangchong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] zhuangchong opened a new pull request, #591: [FLINK-31394] Fix spark jar name in the create release script for table store
zhuangchong opened a new pull request, #591: URL: https://github.com/apache/flink-table-store/pull/591 Fix spark jar name in the create release script for table store -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31394) Fix spark jar name in the create release script for table store
zhuangchong created FLINK-31394: --- Summary: Fix spark jar name in the create release script for table store Key: FLINK-31394 URL: https://issues.apache.org/jira/browse/FLINK-31394 Project: Flink Issue Type: Bug Components: Table Store Affects Versions: table-store-0.4.0 Reporter: zhuangchong -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #584: [FLINK-31338] support infer parallelism for flink table store
JingsongLi commented on code in PR #584:
URL: https://github.com/apache/flink-table-store/pull/584#discussion_r1132003909
##
flink-table-store-flink/flink-table-store-flink-common/src/main/java/org/apache/flink/table/store/connector/source/TableStoreSource.java:
##
@@ -162,15 +164,49 @@ public ScanRuntimeProvider
getScanRuntimeProvider(ScanContext scanContext) {
.withProjection(projectFields)
.withPredicate(predicate)
.withLimit(limit)
-.withParallelism(
-Options.fromMap(table.schema().options())
-
.get(FlinkConnectorOptions.SCAN_PARALLELISM))
+.withParallelism(inferParallelism(table, predicate,
limit, streaming))
.withWatermarkStrategy(watermarkStrategy);
return new TableStoreDataStreamScanProvider(
!streaming, env -> sourceBuilder.withEnv(env).build());
}
+private Integer inferParallelism(
+FileStoreTable table, Predicate predicate, Long limitCount,
boolean streaming) {
+Options options = Options.fromMap(this.table.schema().options());
+Integer parallelism = null;
+
+// for streaming mode, set the default parallelism to the bucket
number.
+if (streaming) {
+parallelism = options.get(CoreOptions.BUCKET);
+}
+
+if (options.containsKey(FlinkConnectorOptions.SCAN_PARALLELISM.key()))
{
+parallelism = options.get(FlinkConnectorOptions.SCAN_PARALLELISM);
+}
+
+// batch mode
+if (options.get(FlinkConnectorOptions.INFER_SCAN_PARALLELISM) &&
!streaming) {
Review Comment:
How about:
```
if (options.get(FlinkConnectorOptions.INFER_SCAN_PARALLELISM)) {
if (streaming) {
parallelism = options.get(CoreOptions.BUCKET);
} else {
// infer parallelism for batch
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-31393) HsFileDataManager use an incorrect default timeout
Weijie Guo created FLINK-31393:
--
Summary: HsFileDataManager use an incorrect default timeout
Key: FLINK-31393
URL: https://issues.apache.org/jira/browse/FLINK-31393
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.16.1, 1.17.0
Reporter: Weijie Guo
Assignee: Weijie Guo
For batch shuffle(i.e. hybrid shuffle & sort-merge shuffle), If there is a
fierce contention of the batch shuffle read memory, it will throw a
{{TimeoutException}} to fail downstream task to release memory. But for hybrid
shuffle, It uses an incorrect default timeout(5ms), this will make the job very
easy to fail.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
xuzhiwen1255 commented on PR #22010: URL: https://github.com/apache/flink/pull/22010#issuecomment-1463322542 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698746#comment-17698746
]
xzw0223 commented on FLINK-31378:
-
[~Weijie Guo] No problem.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Assignee: Wencong Liu
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo reassigned FLINK-31378:
--
Assignee: Wencong Liu
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Assignee: Wencong Liu
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698744#comment-17698744
]
Weijie Guo commented on FLINK-31378:
[~xzw0223] Never mind. If you like, you can participate in the review together.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698743#comment-17698743
]
xzw0223 commented on FLINK-31378:
-
[~Weijie Guo] Sorry, I made a mistake,the test introduces the clients
dependency.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] ZhendongBai closed pull request #17599: [FLINK-24688][yarn] update yarn.application-attempt-failures-validity-interval with a available link
ZhendongBai closed pull request #17599: [FLINK-24688][yarn] update yarn.application-attempt-failures-validity-interval with a available link URL: https://github.com/apache/flink/pull/17599 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698741#comment-17698741
]
Weijie Guo commented on FLINK-31378:
[~xzw0223] What version of Flink do you use? I think {{flink-clients}} is
necessary for local execution after FLINK-15090.
In addition, why do you think that the {{flink-runtime}} dependency needs to be
included in the user's pom, doesn't it need to be modified?
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-benchmarks] lindong28 commented on a diff in pull request #66: [FLINK-29825] Replace median-based regression detect algorithm with max/min-based
lindong28 commented on code in PR #66:
URL: https://github.com/apache/flink-benchmarks/pull/66#discussion_r1131966898
##
regression_report_v2.py:
##
@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import urllib
+import urllib2
+
+from regression_report import loadBenchmarkNames
+
+"""
+This is a regression detection algorithm based on the historical
maximum/minimum value, please refer
+to
https://docs.google.com/document/d/1Bvzvq79Ll5yxd1UtC0YzczgFbZPAgPcN3cI0MjVkIag/edit
the detailed design.
+"""
+
+DEFAULT_CODESPEED_URL = 'http://codespeed.dak8s.net:8000/'
+ENVIRONMENT = 2
+DEFAULT_THRESHOLD = 0.04
+DEFAULT_BASELINE = 30
+
+"""
+Returns a list of benchmark results
+"""
+def loadHistoryData(codespeedUrl, exe, benchmark, downloadSamples):
+url = codespeedUrl + 'timeline/json/?' + urllib.urlencode({'exe': exe,
'ben': benchmark, 'env': ENVIRONMENT, 'revs': downloadSamples})
+f = urllib2.urlopen(url)
+response = f.read()
+f.close()
+timelines = json.loads(response)['timelines'][0]
+result = timelines['branches']['master'][exe]
+lessIsbBetter = (timelines['lessisbetter'] == " (less is better)")
+return result, lessIsbBetter
+
+def checkWithMax(urlToBenchmark, stds, scores, index, baselineSize):
+sustainable_x = [min(scores[i - 2 : i + 1]) for i in range(index -
baselineSize, index)]
+baseline_throughput = max(sustainable_x)
+current_throughput = max(scores[index - 3 : index])
+current_unstable = stds[index] / current_throughput
+if 1 - current_throughput / baseline_throughput > max(DEFAULT_THRESHOLD, 2
* current_unstable):
+print "<%s|%s> baseline=%s current_value=%s" % (urlToBenchmark,
benchmark, baseline_throughput, current_throughput)
+
+def checkWithMin(urlToBenchmark, stds, scores, index, baselineSize):
+sustainable_x = [max(scores[i - 2 : i + 1]) for i in range(index -
baselineSize, index)]
+baseline_throughput = min(sustainable_x)
+current_throughput = min(scores[index - 3 : index])
+current_unstable = stds[index] / current_throughput
+if 1 - current_throughput / baseline_throughput < -1.0 *
max(DEFAULT_THRESHOLD, 2 * current_unstable):
+print "<%s|%s> baseline=%s current_value=%s" % (urlToBenchmark,
benchmark, baseline_throughput, current_throughput)
+
+def checkBenchmark(args, exe, benchmark):
+results, lessIsbBetter = loadHistoryData(args.codespeed, exe, benchmark,
args.downloadSamples)
+results = list(reversed(results))
+scores = [score for (date, score, deviation, commit, branch) in results]
+stds = [deviation for (date, score, deviation, commit, branch) in results]
+
+urlToBenchmark = args.codespeed + 'timeline/#/?' + urllib.urlencode({
+'ben': benchmark,
+'exe': exe,
+'env': ENVIRONMENT,
+'revs': args.downloadSamples,
+'equid': 'off',
+'quarts': 'on',
+'extr': 'on'})
+
+if len(results) < args.baseLine:
+return
+
+if lessIsbBetter:
+checkWithMin(urlToBenchmark, stds, scores, len(scores) - 1,
args.baseLine)
+else:
+checkWithMax(urlToBenchmark, stds, scores, len(scores) - 1,
args.baseLine)
+
+
+if __name__ == "__main__":
+parser = argparse.ArgumentParser(description='Regression report based on
Max/Min value')
+parser.add_argument('--base-line-size', dest='baseLine', required=False,
default=DEFAULT_BASELINE, type=int,
+help='Number of samples taken as the base line.')
+parser.add_argument('--download-samples-size', dest='downloadSamples',
required=False, default=200,
Review Comment:
I see. Thanks for the explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
[GitHub] [flink-benchmarks] lindong28 commented on a diff in pull request #66: [FLINK-29825] Replace median-based regression detect algorithm with max/min-based
lindong28 commented on code in PR #66:
URL: https://github.com/apache/flink-benchmarks/pull/66#discussion_r1131955641
##
regression_report_v2.py:
##
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import urllib
+import urllib2
+
+from regression_report import loadBenchmarkNames
+
+"""
+This is a regression detection algorithm based on the historical
maximum/minimum value, please refer
+to
https://docs.google.com/document/d/1Bvzvq79Ll5yxd1UtC0YzczgFbZPAgPcN3cI0MjVkIag/edit
the detailed design.
+"""
+
+ENVIRONMENT = 2
+
+"""
+Returns a list of benchmark results
+"""
+def loadHistoryData(codespeedUrl, exe, benchmark, baselineSize):
+url = codespeedUrl + 'timeline/json/?' + urllib.urlencode(
+{'exe': exe, 'ben': benchmark, 'env': ENVIRONMENT, 'revs':
baselineSize})
+f = urllib2.urlopen(url)
+response = f.read()
+f.close()
+timelines = json.loads(response)['timelines'][0]
+result = timelines['branches']['master'][exe]
+lessIsbBetter = (timelines['lessisbetter'] == " (less is better)")
+return result, lessIsbBetter
+
+def detectRegression(urlToBenchmark, stds, scores, baselineSize,
minRegressionRatio, minInstabilityMultiplier,
+ direction):
+sustainable_x = [min(scores[i - 3: i]) for i in range(3, baselineSize)]
+baseline_throughput = max(sustainable_x)
+current_throughput = max(scores[-3:])
+current_instability = stds[-1] / current_throughput
+if direction * (1 - current_throughput / baseline_throughput) >
max(minRegressionRatio, minInstabilityMultiplier * current_instability):
+print "<%s|%s> baseline=%s current_value=%s" % (urlToBenchmark,
benchmark, baseline_throughput, current_throughput)
+
+def checkBenchmark(args, exe, benchmark):
+results, lessIsbBetter = loadHistoryData(args.codespeedUrl, exe,
benchmark, args.baselineSize + 3)
+results = list(reversed(results))
+scores = [score for (date, score, deviation, commit, branch) in results]
+stds = [deviation for (date, score, deviation, commit, branch) in results]
+
+urlToBenchmark = args.codespeedUrl + 'timeline/#/?' + urllib.urlencode({
+'ben': benchmark,
+'exe': exe,
+'env': ENVIRONMENT,
+'revs': args.displaySamples,
+'equid': 'off',
+'quarts': 'on',
+'extr': 'on'})
+
+if len(results) < args.baselineSize:
Review Comment:
Does this mean that a new benchmark will only be included in the regression
detection 30 days after this benchmark is added?
I am wondering if it would be useful to include new benchmark in the
detection faster. For example, we can start to detect regression after there
are 5 samples for this benchmark.
##
regression_report_v2.py:
##
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import urllib
+import urllib2
+
+from regression_report import loadBenchmarkNames
+
+"""
+This is a regression detection
[GitHub] [flink-benchmarks] lindong28 commented on a diff in pull request #66: [FLINK-29825] Replace median-based regression detect algorithm with max/min-based
lindong28 commented on code in PR #66:
URL: https://github.com/apache/flink-benchmarks/pull/66#discussion_r1131954846
##
regression_report_v2.py:
##
@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import urllib
+import urllib2
+
+from regression_report import loadBenchmarkNames
+
+"""
+This is a regression detection algorithm based on the historical
maximum/minimum value, please refer
+to
https://docs.google.com/document/d/1Bvzvq79Ll5yxd1UtC0YzczgFbZPAgPcN3cI0MjVkIag/edit
the detailed design.
+"""
+
+DEFAULT_CODESPEED_URL = 'http://codespeed.dak8s.net:8000/'
+ENVIRONMENT = 2
+DEFAULT_THRESHOLD = 0.04
+DEFAULT_BASELINE = 30
+
+"""
+Returns a list of benchmark results
+"""
+def loadHistoryData(codespeedUrl, exe, benchmark, downloadSamples):
+url = codespeedUrl + 'timeline/json/?' + urllib.urlencode({'exe': exe,
'ben': benchmark, 'env': ENVIRONMENT, 'revs': downloadSamples})
+f = urllib2.urlopen(url)
+response = f.read()
+f.close()
+timelines = json.loads(response)['timelines'][0]
+result = timelines['branches']['master'][exe]
+lessIsbBetter = (timelines['lessisbetter'] == " (less is better)")
+return result, lessIsbBetter
+
+def checkWithMax(urlToBenchmark, stds, scores, index, baselineSize):
+sustainable_x = [min(scores[i - 2 : i + 1]) for i in range(index -
baselineSize, index)]
+baseline_throughput = max(sustainable_x)
+current_throughput = max(scores[index - 3 : index])
+current_unstable = stds[index] / current_throughput
+if 1 - current_throughput / baseline_throughput > max(DEFAULT_THRESHOLD, 2
* current_unstable):
+print "<%s|%s> baseline=%s current_value=%s" % (urlToBenchmark,
benchmark, baseline_throughput, current_throughput)
+
+def checkWithMin(urlToBenchmark, stds, scores, index, baselineSize):
+sustainable_x = [max(scores[i - 2 : i + 1]) for i in range(index -
baselineSize, index)]
+baseline_throughput = min(sustainable_x)
+current_throughput = min(scores[index - 3 : index])
+current_unstable = stds[index] / current_throughput
+if 1 - current_throughput / baseline_throughput < -1.0 *
max(DEFAULT_THRESHOLD, 2 * current_unstable):
+print "<%s|%s> baseline=%s current_value=%s" % (urlToBenchmark,
benchmark, baseline_throughput, current_throughput)
+
+def checkBenchmark(args, exe, benchmark):
+results, lessIsbBetter = loadHistoryData(args.codespeed, exe, benchmark,
args.downloadSamples)
+results = list(reversed(results))
+scores = [score for (date, score, deviation, commit, branch) in results]
+stds = [deviation for (date, score, deviation, commit, branch) in results]
+
+urlToBenchmark = args.codespeed + 'timeline/#/?' + urllib.urlencode({
+'ben': benchmark,
+'exe': exe,
+'env': ENVIRONMENT,
+'revs': args.downloadSamples,
+'equid': 'off',
+'quarts': 'on',
+'extr': 'on'})
+
+if len(results) < args.baseLine:
+return
+
+if lessIsbBetter:
+checkWithMin(urlToBenchmark, stds, scores, len(scores) - 1,
args.baseLine)
+else:
+checkWithMax(urlToBenchmark, stds, scores, len(scores) - 1,
args.baseLine)
+
+
+if __name__ == "__main__":
+parser = argparse.ArgumentParser(description='Regression report based on
Max/Min value')
+parser.add_argument('--base-line-size', dest='baseLine', required=False,
default=DEFAULT_BASELINE, type=int,
Review Comment:
I see. Thanks for the explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-31389) Fix spark jar name in docs for table store
[ https://issues.apache.org/jira/browse/FLINK-31389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31389. Assignee: zhuangchong Resolution: Fixed master: 0ffa6654b2d64fc65c430e453e656fa68ce74632 > Fix spark jar name in docs for table store > -- > > Key: FLINK-31389 > URL: https://issues.apache.org/jira/browse/FLINK-31389 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: zhuangchong >Assignee: zhuangchong >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31389) Fix spark jar name in docs for table store
[ https://issues.apache.org/jira/browse/FLINK-31389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31389: --- Labels: pull-request-available (was: ) > Fix spark jar name in docs for table store > -- > > Key: FLINK-31389 > URL: https://issues.apache.org/jira/browse/FLINK-31389 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: zhuangchong >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #588: [FLINK-31389] Fix spark jar name in docs for table store
JingsongLi merged PR #588: URL: https://github.com/apache/flink-table-store/pull/588 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31391) Introduce lookup changelog producer
[ https://issues.apache.org/jira/browse/FLINK-31391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-31391: - Description: Currently, only full-compaction can produce changelog, some merge-engine must have changelog producing, for example, partial-update and aggregation. But full-compaction is very heavy, write amplification is big huge... We should introduce a new changelog producer, supports lower latency to produce changelog. POC: https://github.com/apache/flink-table-store/pull/590 was: Currently, only full-compaction can produce changelog, some merge-engine must have changelog producing, for example, partial-update and aggregation. But full-compaction is very heavy, write amplification is big huge... We should introduce a new changelog producer, supports lower latency to produce changelog. > Introduce lookup changelog producer > --- > > Key: FLINK-31391 > URL: https://issues.apache.org/jira/browse/FLINK-31391 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, only full-compaction can produce changelog, some merge-engine must > have changelog producing, for example, partial-update and aggregation. But > full-compaction is very heavy, write amplification is big huge... > We should introduce a new changelog producer, supports lower latency to > produce changelog. > > POC: https://github.com/apache/flink-table-store/pull/590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31392) Refactor classes code of full-compaction
[ https://issues.apache.org/jira/browse/FLINK-31392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31392: --- Labels: pull-request-available (was: ) > Refactor classes code of full-compaction > > > Key: FLINK-31392 > URL: https://issues.apache.org/jira/browse/FLINK-31392 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > Refactor classes code of full-compaction, this is to prepare some shared > codes for lookup changelog producer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi opened a new pull request, #589: [FLINK-31392] Refactor classes code of full-compaction
JingsongLi opened a new pull request, #589: URL: https://github.com/apache/flink-table-store/pull/589 Refactor classes code of full-compaction, this is to prepare some shared codes for lookup changelog producer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31392) Refactor classes code of full-compaction
Jingsong Lee created FLINK-31392: Summary: Refactor classes code of full-compaction Key: FLINK-31392 URL: https://issues.apache.org/jira/browse/FLINK-31392 Project: Flink Issue Type: Sub-task Components: Table Store Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: table-store-0.4.0 Refactor classes code of full-compaction, this is to prepare some shared codes for lookup changelog producer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31391) Introduce lookup changelog producer
Jingsong Lee created FLINK-31391: Summary: Introduce lookup changelog producer Key: FLINK-31391 URL: https://issues.apache.org/jira/browse/FLINK-31391 Project: Flink Issue Type: New Feature Components: Table Store Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: table-store-0.4.0 Currently, only full-compaction can produce changelog, some merge-engine must have changelog producing, for example, partial-update and aggregation. But full-compaction is very heavy, write amplification is big huge... We should introduce a new changelog producer, supports lower latency to produce changelog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698727#comment-17698727
]
Hongshun Wang commented on FLINK-31378:
---
[~Wencong Liu] , Of course, I’d like to participate in it.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698726#comment-17698726
]
Wencong Liu commented on FLINK-31378:
-
[~loserwang1024] | have a fix for this document. If you like, you can
participate in the review together.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698725#comment-17698725
]
Weijie Guo commented on FLINK-31378:
[~loserwang1024] Yes, add document for beginners sounds good to me. But before
starting this work, we should fix the error in
`https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging`
first.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698722#comment-17698722
]
Hongshun Wang commented on FLINK-31378:
---
[~Weijie Guo] ,if add this in corresponding documentation, it's will be better
for beginners like me.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #22145: [FLINK-31390][table-planner] use scala view to avoid unnecessary trav…
flinkbot commented on PR #22145: URL: https://github.com/apache/flink/pull/22145#issuecomment-1463217141 ## CI report: * 32b4fa1f4d563a407dc36d1d7e6b2e26bb889f61 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698721#comment-17698721
]
xzw0223 commented on FLINK-31378:
-
I think there is no problem with the documentation, and I can execute it
according to the documentation
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
liuyongvs commented on PR #22144: URL: https://github.com/apache/flink/pull/22144#issuecomment-1463213377 hi @snuyanzin @twalthr after talking in https://github.com/apache/flink/pull/21947 1) do not support array_size, which can be replaced by CARDINALITY 2) the select array_remove(array[1, 2], cast(null as int)); throw exception, which is fixed and merge in https://github.com/apache/flink/commit/10dce7cf0a04b80d7416a5760e1a6dbc430d9f88 3) the java equals may cause bug in array_distinct, should use ExpressionEvaluator, suggested by @twalthr which is fixed and merge in https://github.com/apache/flink/commit/6797d6f2592373b2606ddd8c8aad316d677c1cc6 4) the last problem found in https://github.com/apache/flink/pull/22143#issuecomment-1463191208, will also need to be solved so it is the time to implment other array functions. so i submit a new PR, could you help to review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698720#comment-17698720
]
Weijie Guo edited comment on FLINK-31378 at 3/10/23 3:52 AM:
-
If you want to run flink job in the IDE, the dependency of {{flink-clients}}
need to be included in your pom.xml. In case of Table API programs, you will
also need {{flink-table-runtime}} and {{{}flink-table-planner-loader{}}}.
In general, we should mark these dependencies to `provided` scope. As a result,
to make the applications run within IntelliJ IDEA, it is necessary to tick the
{{Include dependencies with "Provided" scope}} box in the run configuration. If
this option is not available (possibly due to using an older IntelliJ IDEA
version), then a workaround is to create a test that calls the application’s
{{main()}} method.
was (Author: weijie guo):
If you want to run flink job in the IDE, the dependency of {{flink-clients}}
need to be included in your pom.xml. In case of Table API programs, you will
also need {{flink-table-runtime}} and {{flink-table-planner-loader}}.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31390) Optimize the FlinkChangelogModeInferenceProgram by avoiding unnecessary traversals.
[
https://issues.apache.org/jira/browse/FLINK-31390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31390:
---
Labels: pull-request-available (was: )
> Optimize the FlinkChangelogModeInferenceProgram by avoiding unnecessary
> traversals.
> ---
>
> Key: FLINK-31390
> URL: https://issues.apache.org/jira/browse/FLINK-31390
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: Aitozi
>Priority: Minor
> Labels: pull-request-available
>
> We can avoid the unnecessary traversals of the RelNode tree, since we are
> only interested in the first satisfied plan.
>
> FlinkChangelogModeInferenceProgram
> {code:java}
> val updateKindTraitVisitor = new SatisfyUpdateKindTraitVisitor(context)
> val finalRoot = requiredUpdateKindTraits.flatMap {
> requiredUpdateKindTrait =>
> updateKindTraitVisitor.visit(rootWithModifyKindSet,
> requiredUpdateKindTrait)
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] Aitozi opened a new pull request, #22145: [FLINK-31390][table-planner] use scala view to avoid unnecessary trav…
Aitozi opened a new pull request, #22145: URL: https://github.com/apache/flink/pull/22145 …ersal of RelNode tree ## What is the purpose of the change Use lazy view to avoid traverse the RelNode tree twice in some case. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698720#comment-17698720
]
Weijie Guo edited comment on FLINK-31378 at 3/10/23 3:49 AM:
-
If you want to run flink job in the IDE, the dependency of {{flink-clients need
to be included in your pom.xml. In case of Table API programs, you will also
need {{flink-table-runtime and {{{}flink-table-planner-loader{}}}.
was (Author: weijie guo):
If you want to run flink job in the IDE, the dependency of {{flink-clients
}}need to be included in your pom.xml. In case of Table API programs, you will
also need {{{}flink-table-runtime{}}}{{{}{}}} and
{{{}flink-table-planner-loader{}}}. {{}}
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698720#comment-17698720
]
Weijie Guo edited comment on FLINK-31378 at 3/10/23 3:49 AM:
-
If you want to run flink job in the IDE, the dependency of {{flink-clients}}
need to be included in your pom.xml. In case of Table API programs, you will
also need {{flink-table-runtime}} and {{flink-table-planner-loader}}.
was (Author: weijie guo):
If you want to run flink job in the IDE, the dependency of {{flink-clients need
to be included in your pom.xml. In case of Table API programs, you will also
need {{flink-table-runtime and {{{}flink-table-planner-loader{}}}.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698720#comment-17698720
]
Weijie Guo commented on FLINK-31378:
If you want to run flink job in the IDE, the dependency of {{flink-clients
}}need to be included in your pom.xml. In case of Table API programs, you will
also need {{{}flink-table-runtime{}}}{{{}{}}} and
{{{}flink-table-planner-loader{}}}. {{}}
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
flinkbot commented on PR #22144: URL: https://github.com/apache/flink/pull/22144#issuecomment-1463210357 ## CI report: * 603c5f92ab0c367e0ccc0ea753fa810a8da06b8f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698718#comment-17698718
]
Hongshun Wang commented on FLINK-31378:
---
Ok, [~xzw0223] , you just write any table api programs or stream programs
(flink version is 1.16.0) without
flink-clients dependency, then run by simply executing the main class rather
than flink cluster. The problem can reproduce.
In fact, it's not a problem or bug , just information lacks in official
tutorials.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31102) Add ARRAY_REMOVE supported in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-31102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31102:
---
Labels: pull-request-available (was: )
> Add ARRAY_REMOVE supported in SQL & Table API
> -
>
> Key: FLINK-31102
> URL: https://issues.apache.org/jira/browse/FLINK-31102
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Remove all elements that equal to element from array.
> Syntax:
> array_remove(array)
> Arguments:
> array: An ARRAY to be handled.
> Returns:
> An ARRAY. If value is NULL, the result is NULL.
> Examples:
> {code:sql}
> SELECT array_remove(array(1, 2, 3, null, 3), 3);
> -- [1,2,null]
> {code}
> See also
> spark
> [[https://spark.apache.org/docs/latest/api/sql/index.html#array_size]|https://spark.apache.org/docs/latest/api/sql/index.html#array_remove]
> presto [https://prestodb.io/docs/current/functions/array.html]
> postgresql
> [https://w3resource.com/PostgreSQL/postgresql_array_remove-function.php]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] liuyongvs opened a new pull request, #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
liuyongvs opened a new pull request, #22144: URL: https://github.com/apache/flink/pull/22144 - What is the purpose of the change This is an implementation of ARRAY_REMOVE - Brief change log ARRAY_SIZE for Table API and SQL ``` Syntax: array_remove(array) Arguments: array: An ARRAY to be handled. Returns: An ARRAY. If value is NULL, the result is NULL. Examples: SELECT array_remove(array(1, 2, 3, null, 3), 3); -- [1,2,null] ``` See also spark [[https://spark.apache.org/docs/latest/api/sql/index.html#array_size]](https://spark.apache.org/docs/latest/api/sql/index.html#array_remove) presto https://prestodb.io/docs/current/functions/array.html postgresql https://w3resource.com/PostgreSQL/postgresql_array_remove-function.php - Verifying this change This change added tests in CollectionFunctionsITCase - Does this pull request potentially affect one of the following parts: Dependencies (does it add or upgrade a dependency): ( no) The public API, i.e., is any changed class annotated with @Public(Evolving): (yes ) The serializers: (no) The runtime per-record code paths (performance sensitive): ( no) Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no) The S3 file system connector: ( no) - Documentation Does this pull request introduce a new feature? (yes) If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698716#comment-17698716
]
xzw0223 commented on FLINK-31378:
-
I think you can upload your pom and describe more details, it will be easier
for me to reproduce the problem.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698713#comment-17698713
]
Hongshun Wang commented on FLINK-31378:
---
[~xzw0223] , I have already tried table api without flink-clients. It shows
that same error is still thrown.
!image-2023-03-10-11-19-35-773.png!
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongshun Wang updated FLINK-31378:
--
Attachment: image-2023-03-10-11-19-35-773.png
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #22143: [FLINK-31377][table] Fix array_contains ArrayData.ElementGetter shoul…
flinkbot commented on PR #22143: URL: https://github.com/apache/flink/pull/22143#issuecomment-1463191208 ## CI report: * 2a2f0a9392c16f8d52dea8afcad66e5b13015ffd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31377) BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0
[
https://issues.apache.org/jira/browse/FLINK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698710#comment-17698710
]
jackylau commented on FLINK-31377:
--
hi [~snuyanzin] , array_contains have another bug, could you also have a look.
> BinaryArrayData getArray/getMap should Handle null correctly AssertionError:
> valueArraySize (-6) should >= 0
> -
>
> Key: FLINK-31377
> URL: https://issues.apache.org/jira/browse/FLINK-31377
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
>
> you can reproduce this error below. and reason is in ARRAY_CONTAINS
> {code:java}
> if the needle is a Map NOT NULL,and the array has null element.
> this bellowing will cause getElementOrNull(ArrayData array, int pos) only can
> handle not null. so it throw exception
> /*elementGetter =
> ArrayData.createElementGetter(needleDataType.getLogicalType());*/,
> {code}
>
> {code:java}
> // code placeholder
> Stream getTestSetSpecs() {
> return Stream.of(
> TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
> .onFieldsWithData(
> new Map[] {
> null,
> CollectionUtil.map(entry(1, "a"), entry(2,
> "b")),
> CollectionUtil.map(entry(3, "c"), entry(4,
> "d")),
> },
> null)
> .andDataTypes(
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
> DataTypes.STRING())),
> DataTypes.STRING())
> .testResult(
> $("f0").arrayContains(
> CollectionUtil.map(entry(3, "c"),
> entry(4, "d"))),
> "ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
> true,
> DataTypes.BOOLEAN()));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31155) Build and stage Java and Python artifacts
[
https://issues.apache.org/jira/browse/FLINK-31155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698708#comment-17698708
]
Qingsheng Ren edited comment on FLINK-31155 at 3/10/23 3:09 AM:
* [Key|https://dist.apache.org/repos/dist/release/flink/KEYS] fingerprint:
A1BD477F79D036D2C30CA7DBCA8AEEC2F6EB040B
* Maven central deployment:
[https://repository.apache.org/content/repositories/orgapacheflink-1591]
* Tag {{{}release-1.17.0-rc1{}}}:
[https://github.com/apache/flink/releases/tag/release-1.17.0-rc1]
was (Author: renqs):
* Source and binary:
[https://dist.apache.org/repos/dist/dev/flink/flink-1.17.0-rc1/]
* [Key|https://dist.apache.org/repos/dist/release/flink/KEYS] fingerprint:
A1BD477F79D036D2C30CA7DBCA8AEEC2F6EB040B
* Maven central deployment:
[https://repository.apache.org/content/repositories/orgapacheflink-1591]
* Tag {{{}release-1.17.0-rc1{}}}:
[https://github.com/apache/flink/releases/tag/release-1.17.0-rc1]
> Build and stage Java and Python artifacts
> -
>
> Key: FLINK-31155
> URL: https://issues.apache.org/jira/browse/FLINK-31155
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> # Create a local release branch ((!) this step can not be skipped for minor
> releases):
> {code:bash}
> $ cd ./tools
> tools/ $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION
> RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh
> {code}
> # Tag the release commit:
> {code:bash}
> $ git tag -s ${TAG} -m "${TAG}"
> {code}
> # We now need to do several things:
> ## Create the source release archive
> ## Deploy jar artefacts to the [Apache Nexus
> Repository|https://repository.apache.org/], which is the staging area for
> deploying the jars to Maven Central
> ## Build PyFlink wheel packages
> You might want to create a directory on your local machine for collecting the
> various source and binary releases before uploading them. Creating the binary
> releases is a lengthy process but you can do this on another machine (for
> example, in the "cloud"). When doing this, you can skip signing the release
> files on the remote machine, download them to your local machine and sign
> them there.
> # Build the source release:
> {code:bash}
> tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
> {code}
> # Stage the maven artifacts:
> {code:bash}
> tools $ releasing/deploy_staging_jars.sh
> {code}
> Review all staged artifacts ([https://repository.apache.org/]). They should
> contain all relevant parts for each module, including pom.xml, jar, test jar,
> source, test source, javadoc, etc. Carefully review any new artifacts.
> # Close the staging repository on Apache Nexus. When prompted for a
> description, enter “Apache Flink, version X, release candidate Y”.
> Then, you need to build the PyFlink wheel packages (since 1.11):
> # Set up an azure pipeline in your own Azure account. You can refer to
> [Azure
> Pipelines|https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository]
> for more details on how to set up azure pipeline for a fork of the Flink
> repository. Note that a google cloud mirror in Europe is used for downloading
> maven artifacts, therefore it is recommended to set your [Azure organization
> region|https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/change-organization-location]
> to Europe to speed up the downloads.
> # Push the release candidate branch to your forked personal Flink
> repository, e.g.
> {code:bash}
> tools $ git push
> refs/heads/release-${RELEASE_VERSION}-rc${RC_NUM}:release-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
> # Trigger the Azure Pipelines manually to build the PyFlink wheel packages
> ## Go to your Azure Pipelines Flink project → Pipelines
> ## Click the "New pipeline" button on the top right
> ## Select "GitHub" → your GitHub Flink repository → "Existing Azure
> Pipelines YAML file"
> ## Select your branch → Set path to "/azure-pipelines.yaml" → click on
> "Continue" → click on "Variables"
> ## Then click "New Variable" button, fill the name with "MODE", and the
> value with "release". Click "OK" to set the variable and the "Save" button to
> save the variables, then back on the "Review your pipeline" screen click
> "Run" to trigger the build.
> ## You should now see a build where only the "CI build (release)" is running
> # Download the PyFlink wheel packages from the build result page after the
> jobs of "build_wheels mac" and "build_wheels linux" have finished.
> ## Download the PyFlink wheel packages
> ### Open the build result page of the pipeline
> ### Go to the {{Artifacts}} page
[jira] [Resolved] (FLINK-31156) Stage source and binary releases on dist.apache.org
[
https://issues.apache.org/jira/browse/FLINK-31156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren resolved FLINK-31156.
---
Resolution: Done
> Stage source and binary releases on dist.apache.org
> ---
>
> Key: FLINK-31156
> URL: https://issues.apache.org/jira/browse/FLINK-31156
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> Copy the source release to the dev repository of dist.apache.org:
> # If you have not already, check out the Flink section of the dev repository
> on dist.apache.org via Subversion. In a fresh directory:
> {code:bash}
> $ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
> {code}
> # Make a directory for the new release and copy all the artifacts (Flink
> source/binary distributions, hashes, GPG signatures and the python
> subdirectory) into that newly created directory:
> {code:bash}
> $ mkdir flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
> $ mv /tools/releasing/release/*
> flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
> # Add and commit all the files.
> {code:bash}
> $ cd flink
> flink $ svn add flink-${RELEASE_VERSION}-rc${RC_NUM}
> flink $ svn commit -m "Add flink-${RELEASE_VERSION}-rc${RC_NUM}"
> {code}
> # Verify that files are present under
> [https://dist.apache.org/repos/dist/dev/flink|https://dist.apache.org/repos/dist/dev/flink].
> # Push the release tag if not done already (the following command assumes to
> be called from within the apache/flink checkout):
> {code:bash}
> $ git push refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
>
>
> h3. Expectations
> * Maven artifacts deployed to the staging repository of
> [repository.apache.org|https://repository.apache.org/content/repositories/]
> * Source distribution deployed to the dev repository of
> [dist.apache.org|https://dist.apache.org/repos/dist/dev/flink/]
> * Check hashes (e.g. shasum -c *.sha512)
> * Check signatures (e.g. {{{}gpg --verify
> flink-1.2.3-source-release.tar.gz.asc flink-1.2.3-source-release.tar.gz{}}})
> * {{grep}} for legal headers in each file.
> * If time allows check the NOTICE files of the modules whose dependencies
> have been changed in this release in advance, since the license issues from
> time to time pop up during voting. See [Verifying a Flink
> Release|https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release]
> "Checking License" section.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31156) Stage source and binary releases on dist.apache.org
[
https://issues.apache.org/jira/browse/FLINK-31156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698709#comment-17698709
]
Qingsheng Ren commented on FLINK-31156:
---
* Source and binary:
[https://dist.apache.org/repos/dist/dev/flink/flink-1.17.0-rc1/]
* Tag {{{}release-1.17.0-rc1{}}}:
[https://github.com/apache/flink/releases/tag/release-1.17.0-rc1]
> Stage source and binary releases on dist.apache.org
> ---
>
> Key: FLINK-31156
> URL: https://issues.apache.org/jira/browse/FLINK-31156
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>
> Copy the source release to the dev repository of dist.apache.org:
> # If you have not already, check out the Flink section of the dev repository
> on dist.apache.org via Subversion. In a fresh directory:
> {code:bash}
> $ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
> {code}
> # Make a directory for the new release and copy all the artifacts (Flink
> source/binary distributions, hashes, GPG signatures and the python
> subdirectory) into that newly created directory:
> {code:bash}
> $ mkdir flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
> $ mv /tools/releasing/release/*
> flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
> # Add and commit all the files.
> {code:bash}
> $ cd flink
> flink $ svn add flink-${RELEASE_VERSION}-rc${RC_NUM}
> flink $ svn commit -m "Add flink-${RELEASE_VERSION}-rc${RC_NUM}"
> {code}
> # Verify that files are present under
> [https://dist.apache.org/repos/dist/dev/flink|https://dist.apache.org/repos/dist/dev/flink].
> # Push the release tag if not done already (the following command assumes to
> be called from within the apache/flink checkout):
> {code:bash}
> $ git push refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
>
>
> h3. Expectations
> * Maven artifacts deployed to the staging repository of
> [repository.apache.org|https://repository.apache.org/content/repositories/]
> * Source distribution deployed to the dev repository of
> [dist.apache.org|https://dist.apache.org/repos/dist/dev/flink/]
> * Check hashes (e.g. shasum -c *.sha512)
> * Check signatures (e.g. {{{}gpg --verify
> flink-1.2.3-source-release.tar.gz.asc flink-1.2.3-source-release.tar.gz{}}})
> * {{grep}} for legal headers in each file.
> * If time allows check the NOTICE files of the modules whose dependencies
> have been changed in this release in advance, since the license issues from
> time to time pop up during voting. See [Verifying a Flink
> Release|https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release]
> "Checking License" section.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-31156) Stage source and binary releases on dist.apache.org
[
https://issues.apache.org/jira/browse/FLINK-31156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren reassigned FLINK-31156:
-
Assignee: Qingsheng Ren
> Stage source and binary releases on dist.apache.org
> ---
>
> Key: FLINK-31156
> URL: https://issues.apache.org/jira/browse/FLINK-31156
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> Copy the source release to the dev repository of dist.apache.org:
> # If you have not already, check out the Flink section of the dev repository
> on dist.apache.org via Subversion. In a fresh directory:
> {code:bash}
> $ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
> {code}
> # Make a directory for the new release and copy all the artifacts (Flink
> source/binary distributions, hashes, GPG signatures and the python
> subdirectory) into that newly created directory:
> {code:bash}
> $ mkdir flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
> $ mv /tools/releasing/release/*
> flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
> # Add and commit all the files.
> {code:bash}
> $ cd flink
> flink $ svn add flink-${RELEASE_VERSION}-rc${RC_NUM}
> flink $ svn commit -m "Add flink-${RELEASE_VERSION}-rc${RC_NUM}"
> {code}
> # Verify that files are present under
> [https://dist.apache.org/repos/dist/dev/flink|https://dist.apache.org/repos/dist/dev/flink].
> # Push the release tag if not done already (the following command assumes to
> be called from within the apache/flink checkout):
> {code:bash}
> $ git push refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
>
>
> h3. Expectations
> * Maven artifacts deployed to the staging repository of
> [repository.apache.org|https://repository.apache.org/content/repositories/]
> * Source distribution deployed to the dev repository of
> [dist.apache.org|https://dist.apache.org/repos/dist/dev/flink/]
> * Check hashes (e.g. shasum -c *.sha512)
> * Check signatures (e.g. {{{}gpg --verify
> flink-1.2.3-source-release.tar.gz.asc flink-1.2.3-source-release.tar.gz{}}})
> * {{grep}} for legal headers in each file.
> * If time allows check the NOTICE files of the modules whose dependencies
> have been changed in this release in advance, since the license issues from
> time to time pop up during voting. See [Verifying a Flink
> Release|https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release]
> "Checking License" section.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31377) BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0
[
https://issues.apache.org/jira/browse/FLINK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31377:
---
Labels: pull-request-available (was: )
> BinaryArrayData getArray/getMap should Handle null correctly AssertionError:
> valueArraySize (-6) should >= 0
> -
>
> Key: FLINK-31377
> URL: https://issues.apache.org/jira/browse/FLINK-31377
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
>
> you can reproduce this error below. and reason is in ARRAY_CONTAINS
> {code:java}
> if the needle is a Map NOT NULL,and the array has null element.
> this bellowing will cause getElementOrNull(ArrayData array, int pos) only can
> handle not null. so it throw exception
> /*elementGetter =
> ArrayData.createElementGetter(needleDataType.getLogicalType());*/,
> {code}
>
> {code:java}
> // code placeholder
> Stream getTestSetSpecs() {
> return Stream.of(
> TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
> .onFieldsWithData(
> new Map[] {
> null,
> CollectionUtil.map(entry(1, "a"), entry(2,
> "b")),
> CollectionUtil.map(entry(3, "c"), entry(4,
> "d")),
> },
> null)
> .andDataTypes(
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
> DataTypes.STRING())),
> DataTypes.STRING())
> .testResult(
> $("f0").arrayContains(
> CollectionUtil.map(entry(3, "c"),
> entry(4, "d"))),
> "ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
> true,
> DataTypes.BOOLEAN()));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] liuyongvs opened a new pull request, #22143: [FLINK-31377][table] Fix array_contains ArrayData.ElementGetter shoul…
liuyongvs opened a new pull request, #22143:
URL: https://github.com/apache/flink/pull/22143
## What is the purpose of the change
*Fix array_contains ArrayData.ElementGetter should use element type instead
of needle type.*
- needle and element type are identical. because " a NOT NULL type can be
stored in NULL type but not vice versa." and after dig the code
TypeInferenceOperandChecker.insertImplicitCasts, you can see here
supportsAvoidingCast.
`
// code placeholder
Stream getTestSetSpecs() {
return Stream.of(
TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_CONTAINS)
.onFieldsWithData(
new Map[] {
null,
CollectionUtil.map(entry(1, "a"), entry(2,
"b")),
CollectionUtil.map(entry(3, "c"), entry(4,
"d")),
},
null)
.andDataTypes(
DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(),
DataTypes.STRING())),
DataTypes.STRING())
.testResult(
$("f0").arrayContains(
CollectionUtil.map(entry(3,
"c"), entry(4, "d"))),
"ARRAY_CONTAINS(f0, MAP[3, 'c', 4, 'd'])",
true,
DataTypes.BOOLEAN()));
}
`
## Verifying this change
- add unit test
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
@Public(Evolving): yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing
- Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs / JavaDocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31155) Build and stage Java and Python artifacts
[
https://issues.apache.org/jira/browse/FLINK-31155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698708#comment-17698708
]
Qingsheng Ren commented on FLINK-31155:
---
* Source and binary:
[https://dist.apache.org/repos/dist/dev/flink/flink-1.17.0-rc1/]
* [Key|https://dist.apache.org/repos/dist/release/flink/KEYS] fingerprint:
A1BD477F79D036D2C30CA7DBCA8AEEC2F6EB040B
* Maven central deployment:
[https://repository.apache.org/content/repositories/orgapacheflink-1591]
* Tag {{{}release-1.17.0-rc1{}}}:
[https://github.com/apache/flink/releases/tag/release-1.17.0-rc1]
> Build and stage Java and Python artifacts
> -
>
> Key: FLINK-31155
> URL: https://issues.apache.org/jira/browse/FLINK-31155
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> # Create a local release branch ((!) this step can not be skipped for minor
> releases):
> {code:bash}
> $ cd ./tools
> tools/ $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION
> RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh
> {code}
> # Tag the release commit:
> {code:bash}
> $ git tag -s ${TAG} -m "${TAG}"
> {code}
> # We now need to do several things:
> ## Create the source release archive
> ## Deploy jar artefacts to the [Apache Nexus
> Repository|https://repository.apache.org/], which is the staging area for
> deploying the jars to Maven Central
> ## Build PyFlink wheel packages
> You might want to create a directory on your local machine for collecting the
> various source and binary releases before uploading them. Creating the binary
> releases is a lengthy process but you can do this on another machine (for
> example, in the "cloud"). When doing this, you can skip signing the release
> files on the remote machine, download them to your local machine and sign
> them there.
> # Build the source release:
> {code:bash}
> tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
> {code}
> # Stage the maven artifacts:
> {code:bash}
> tools $ releasing/deploy_staging_jars.sh
> {code}
> Review all staged artifacts ([https://repository.apache.org/]). They should
> contain all relevant parts for each module, including pom.xml, jar, test jar,
> source, test source, javadoc, etc. Carefully review any new artifacts.
> # Close the staging repository on Apache Nexus. When prompted for a
> description, enter “Apache Flink, version X, release candidate Y”.
> Then, you need to build the PyFlink wheel packages (since 1.11):
> # Set up an azure pipeline in your own Azure account. You can refer to
> [Azure
> Pipelines|https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository]
> for more details on how to set up azure pipeline for a fork of the Flink
> repository. Note that a google cloud mirror in Europe is used for downloading
> maven artifacts, therefore it is recommended to set your [Azure organization
> region|https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/change-organization-location]
> to Europe to speed up the downloads.
> # Push the release candidate branch to your forked personal Flink
> repository, e.g.
> {code:bash}
> tools $ git push
> refs/heads/release-${RELEASE_VERSION}-rc${RC_NUM}:release-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
> # Trigger the Azure Pipelines manually to build the PyFlink wheel packages
> ## Go to your Azure Pipelines Flink project → Pipelines
> ## Click the "New pipeline" button on the top right
> ## Select "GitHub" → your GitHub Flink repository → "Existing Azure
> Pipelines YAML file"
> ## Select your branch → Set path to "/azure-pipelines.yaml" → click on
> "Continue" → click on "Variables"
> ## Then click "New Variable" button, fill the name with "MODE", and the
> value with "release". Click "OK" to set the variable and the "Save" button to
> save the variables, then back on the "Review your pipeline" screen click
> "Run" to trigger the build.
> ## You should now see a build where only the "CI build (release)" is running
> # Download the PyFlink wheel packages from the build result page after the
> jobs of "build_wheels mac" and "build_wheels linux" have finished.
> ## Download the PyFlink wheel packages
> ### Open the build result page of the pipeline
> ### Go to the {{Artifacts}} page (build_wheels linux -> 1 artifact)
> ### Click {{wheel_Darwin_build_wheels mac}} and {{wheel_Linux_build_wheels
> linux}} separately to download the zip files
> ## Unzip these two zip files
> {code:bash}
> $ cd /path/to/downloaded_wheel_packages
> $ unzip wheel_Linux_build_wheels\ linux.zip
> $ unzip wheel_Darwin_build_wheels\ mac.zip{code}
> ## Create directory {{./dist}} under the
[jira] [Resolved] (FLINK-31155) Build and stage Java and Python artifacts
[
https://issues.apache.org/jira/browse/FLINK-31155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren resolved FLINK-31155.
---
Resolution: Done
> Build and stage Java and Python artifacts
> -
>
> Key: FLINK-31155
> URL: https://issues.apache.org/jira/browse/FLINK-31155
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> # Create a local release branch ((!) this step can not be skipped for minor
> releases):
> {code:bash}
> $ cd ./tools
> tools/ $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION
> RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh
> {code}
> # Tag the release commit:
> {code:bash}
> $ git tag -s ${TAG} -m "${TAG}"
> {code}
> # We now need to do several things:
> ## Create the source release archive
> ## Deploy jar artefacts to the [Apache Nexus
> Repository|https://repository.apache.org/], which is the staging area for
> deploying the jars to Maven Central
> ## Build PyFlink wheel packages
> You might want to create a directory on your local machine for collecting the
> various source and binary releases before uploading them. Creating the binary
> releases is a lengthy process but you can do this on another machine (for
> example, in the "cloud"). When doing this, you can skip signing the release
> files on the remote machine, download them to your local machine and sign
> them there.
> # Build the source release:
> {code:bash}
> tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
> {code}
> # Stage the maven artifacts:
> {code:bash}
> tools $ releasing/deploy_staging_jars.sh
> {code}
> Review all staged artifacts ([https://repository.apache.org/]). They should
> contain all relevant parts for each module, including pom.xml, jar, test jar,
> source, test source, javadoc, etc. Carefully review any new artifacts.
> # Close the staging repository on Apache Nexus. When prompted for a
> description, enter “Apache Flink, version X, release candidate Y”.
> Then, you need to build the PyFlink wheel packages (since 1.11):
> # Set up an azure pipeline in your own Azure account. You can refer to
> [Azure
> Pipelines|https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository]
> for more details on how to set up azure pipeline for a fork of the Flink
> repository. Note that a google cloud mirror in Europe is used for downloading
> maven artifacts, therefore it is recommended to set your [Azure organization
> region|https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/change-organization-location]
> to Europe to speed up the downloads.
> # Push the release candidate branch to your forked personal Flink
> repository, e.g.
> {code:bash}
> tools $ git push
> refs/heads/release-${RELEASE_VERSION}-rc${RC_NUM}:release-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
> # Trigger the Azure Pipelines manually to build the PyFlink wheel packages
> ## Go to your Azure Pipelines Flink project → Pipelines
> ## Click the "New pipeline" button on the top right
> ## Select "GitHub" → your GitHub Flink repository → "Existing Azure
> Pipelines YAML file"
> ## Select your branch → Set path to "/azure-pipelines.yaml" → click on
> "Continue" → click on "Variables"
> ## Then click "New Variable" button, fill the name with "MODE", and the
> value with "release". Click "OK" to set the variable and the "Save" button to
> save the variables, then back on the "Review your pipeline" screen click
> "Run" to trigger the build.
> ## You should now see a build where only the "CI build (release)" is running
> # Download the PyFlink wheel packages from the build result page after the
> jobs of "build_wheels mac" and "build_wheels linux" have finished.
> ## Download the PyFlink wheel packages
> ### Open the build result page of the pipeline
> ### Go to the {{Artifacts}} page (build_wheels linux -> 1 artifact)
> ### Click {{wheel_Darwin_build_wheels mac}} and {{wheel_Linux_build_wheels
> linux}} separately to download the zip files
> ## Unzip these two zip files
> {code:bash}
> $ cd /path/to/downloaded_wheel_packages
> $ unzip wheel_Linux_build_wheels\ linux.zip
> $ unzip wheel_Darwin_build_wheels\ mac.zip{code}
> ## Create directory {{./dist}} under the directory of {{{}flink-python{}}}:
> {code:bash}
> $ cd
> $ mkdir flink-python/dist{code}
> ## Move the unzipped wheel packages to the directory of
> {{{}flink-python/dist{}}}:
> {code:java}
> $ mv /path/to/wheel_Darwin_build_wheels\ mac/* flink-python/dist/
> $ mv /path/to/wheel_Linux_build_wheels\ linux/* flink-python/dist/
> $ cd tools{code}
> Finally, we create the binary convenience release files:
> {code:bash}
> tools $
[jira] [Assigned] (FLINK-31155) Build and stage Java and Python artifacts
[
https://issues.apache.org/jira/browse/FLINK-31155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren reassigned FLINK-31155:
-
Assignee: Qingsheng Ren
> Build and stage Java and Python artifacts
> -
>
> Key: FLINK-31155
> URL: https://issues.apache.org/jira/browse/FLINK-31155
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> # Create a local release branch ((!) this step can not be skipped for minor
> releases):
> {code:bash}
> $ cd ./tools
> tools/ $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION
> RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh
> {code}
> # Tag the release commit:
> {code:bash}
> $ git tag -s ${TAG} -m "${TAG}"
> {code}
> # We now need to do several things:
> ## Create the source release archive
> ## Deploy jar artefacts to the [Apache Nexus
> Repository|https://repository.apache.org/], which is the staging area for
> deploying the jars to Maven Central
> ## Build PyFlink wheel packages
> You might want to create a directory on your local machine for collecting the
> various source and binary releases before uploading them. Creating the binary
> releases is a lengthy process but you can do this on another machine (for
> example, in the "cloud"). When doing this, you can skip signing the release
> files on the remote machine, download them to your local machine and sign
> them there.
> # Build the source release:
> {code:bash}
> tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
> {code}
> # Stage the maven artifacts:
> {code:bash}
> tools $ releasing/deploy_staging_jars.sh
> {code}
> Review all staged artifacts ([https://repository.apache.org/]). They should
> contain all relevant parts for each module, including pom.xml, jar, test jar,
> source, test source, javadoc, etc. Carefully review any new artifacts.
> # Close the staging repository on Apache Nexus. When prompted for a
> description, enter “Apache Flink, version X, release candidate Y”.
> Then, you need to build the PyFlink wheel packages (since 1.11):
> # Set up an azure pipeline in your own Azure account. You can refer to
> [Azure
> Pipelines|https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository]
> for more details on how to set up azure pipeline for a fork of the Flink
> repository. Note that a google cloud mirror in Europe is used for downloading
> maven artifacts, therefore it is recommended to set your [Azure organization
> region|https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/change-organization-location]
> to Europe to speed up the downloads.
> # Push the release candidate branch to your forked personal Flink
> repository, e.g.
> {code:bash}
> tools $ git push
> refs/heads/release-${RELEASE_VERSION}-rc${RC_NUM}:release-${RELEASE_VERSION}-rc${RC_NUM}
> {code}
> # Trigger the Azure Pipelines manually to build the PyFlink wheel packages
> ## Go to your Azure Pipelines Flink project → Pipelines
> ## Click the "New pipeline" button on the top right
> ## Select "GitHub" → your GitHub Flink repository → "Existing Azure
> Pipelines YAML file"
> ## Select your branch → Set path to "/azure-pipelines.yaml" → click on
> "Continue" → click on "Variables"
> ## Then click "New Variable" button, fill the name with "MODE", and the
> value with "release". Click "OK" to set the variable and the "Save" button to
> save the variables, then back on the "Review your pipeline" screen click
> "Run" to trigger the build.
> ## You should now see a build where only the "CI build (release)" is running
> # Download the PyFlink wheel packages from the build result page after the
> jobs of "build_wheels mac" and "build_wheels linux" have finished.
> ## Download the PyFlink wheel packages
> ### Open the build result page of the pipeline
> ### Go to the {{Artifacts}} page (build_wheels linux -> 1 artifact)
> ### Click {{wheel_Darwin_build_wheels mac}} and {{wheel_Linux_build_wheels
> linux}} separately to download the zip files
> ## Unzip these two zip files
> {code:bash}
> $ cd /path/to/downloaded_wheel_packages
> $ unzip wheel_Linux_build_wheels\ linux.zip
> $ unzip wheel_Darwin_build_wheels\ mac.zip{code}
> ## Create directory {{./dist}} under the directory of {{{}flink-python{}}}:
> {code:bash}
> $ cd
> $ mkdir flink-python/dist{code}
> ## Move the unzipped wheel packages to the directory of
> {{{}flink-python/dist{}}}:
> {code:java}
> $ mv /path/to/wheel_Darwin_build_wheels\ mac/* flink-python/dist/
> $ mv /path/to/wheel_Linux_build_wheels\ linux/* flink-python/dist/
> $ cd tools{code}
> Finally, we create the binary convenience release files:
> {code:bash}
>
