[GitHub] [flink] XComp commented on pull request #22572: [FLINK-29618] Remove Timeout annotation in YARNSessionFIFOSecuredITCase
XComp commented on PR #22572: URL: https://github.com/apache/flink/pull/22572#issuecomment-1545210878 I guess, we could, at least, remove the `@Timeout` from the related `YarnSessionFIFOITCase` as well. WDYT? And what about touching the other tests in `flink-yarn-tests`? Could you check whether it would make sense to remove the timeouts there as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #21971: [FLINK-31084][connectors/dataGen] Add default value for dataGen seque…
XComp commented on code in PR #21971:
URL: https://github.com/apache/flink/pull/21971#discussion_r1191934305
##
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/DataGenTableSource.java:
##
@@ -93,4 +93,9 @@ public ChangelogMode getChangelogMode() {
public void applyLimit(long limit) {
this.numberOfRows = limit;
}
+
+@VisibleForTesting
+public DataGenerator[] getFieldGenerators() {
Review Comment:
Yes, I would suggest that we do that in a separate hotfix commit within this
PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-30629) ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat is unstable
[
https://issues.apache.org/jira/browse/FLINK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722017#comment-17722017
]
Sergey Nuyanzin commented on FLINK-30629:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48940=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=9775
> ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat is unstable
> -
>
> Key: FLINK-30629
> URL: https://issues.apache.org/jira/browse/FLINK-30629
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Xintong Song
>Assignee: Weijie Guo
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
> Attachments: ClientHeartbeatTestLog.txt
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44690=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=10819
> {code:java}
> Jan 11 04:32:39 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 21.02 s <<< FAILURE! - in
> org.apache.flink.client.ClientHeartbeatTest
> Jan 11 04:32:39 [ERROR]
> org.apache.flink.client.ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat
> Time elapsed: 9.157 s <<< ERROR!
> Jan 11 04:32:39 java.lang.IllegalStateException: MiniCluster is not yet
> running or has already been shut down.
> Jan 11 04:32:39 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniCluster.getDispatcherGatewayFuture(MiniCluster.java:1044)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:917)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniCluster.getJobStatus(MiniCluster.java:841)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.getJobStatus(MiniClusterJobClient.java:91)
> Jan 11 04:32:39 at
> org.apache.flink.client.ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat(ClientHeartbeatTest.java:79)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-32054) ElasticsearchSinkITCase.testElasticsearchSink fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-32054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin closed FLINK-32054.
---
Resolution: Won't Fix
> ElasticsearchSinkITCase.testElasticsearchSink fails on AZP
> --
>
> Key: FLINK-32054
> URL: https://issues.apache.org/jira/browse/FLINK-32054
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.16.1
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: test-stability
>
> Test ElasticsearchSinkITCase.testElasticsearchSink fails on AZP
> {noformat}
> May 11 02:00:56 Caused by: org.elasticsearch.client.ResponseException:
> org.elasticsearch.client.ResponseException: method [HEAD], host
> [http://172.17.0.1:50560], URI [/], status line [HTTP/1.1 503 Service
> Unavailable]
> May 11 02:00:56 at
> org.elasticsearch.client.RestClient$1.completed(RestClient.java:552)
> May 11 02:00:56 at
> org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
> May 11 02:00:56 at
> org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
> May 11 02:00:56 at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
> May 11 02:00:56 at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
> May 11 02:00:56 at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:309)
> May 11 02:00:56 at
> org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
> May 11 02:00:56 at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
> May 11 02:00:56 at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> May 11 02:00:56 ... 1 more
> May 11 02:00:56
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48891=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9=15299
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32054) ElasticsearchSinkITCase.testElasticsearchSink fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-32054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722015#comment-17722015
]
Sergey Nuyanzin commented on FLINK-32054:
-
yes, probably you're right at least log the issue
I will close it since ES is now in it's own repo
> ElasticsearchSinkITCase.testElasticsearchSink fails on AZP
> --
>
> Key: FLINK-32054
> URL: https://issues.apache.org/jira/browse/FLINK-32054
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.16.1
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: test-stability
>
> Test ElasticsearchSinkITCase.testElasticsearchSink fails on AZP
> {noformat}
> May 11 02:00:56 Caused by: org.elasticsearch.client.ResponseException:
> org.elasticsearch.client.ResponseException: method [HEAD], host
> [http://172.17.0.1:50560], URI [/], status line [HTTP/1.1 503 Service
> Unavailable]
> May 11 02:00:56 at
> org.elasticsearch.client.RestClient$1.completed(RestClient.java:552)
> May 11 02:00:56 at
> org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
> May 11 02:00:56 at
> org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
> May 11 02:00:56 at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
> May 11 02:00:56 at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
> May 11 02:00:56 at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:309)
> May 11 02:00:56 at
> org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
> May 11 02:00:56 at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
> May 11 02:00:56 at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> May 11 02:00:56 at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> May 11 02:00:56 ... 1 more
> May 11 02:00:56
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48891=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9=15299
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-25909) Move HBase token obtain functionality into HBaseDelegationTokenProvider
[ https://issues.apache.org/jira/browse/FLINK-25909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722009#comment-17722009 ] Lijie Wang commented on FLINK-25909: Change log level to DEBUG: master: 00d4419b1f46fc85c8ef53bd8788002fd7a49bcf release-1.17: 6efc2ac5b56e172cb9061cebef9069e31ae24d90 > Move HBase token obtain functionality into HBaseDelegationTokenProvider > --- > > Key: FLINK-25909 > URL: https://issues.apache.org/jira/browse/FLINK-25909 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.15.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wanglijie95 closed pull request #22559: [hotfix][runtime][security] Change the log level to DEBUG in HBaseDelegationTokenProvider#getHBaseConfiguration
wanglijie95 closed pull request #22559: [hotfix][runtime][security] Change the log level to DEBUG in HBaseDelegationTokenProvider#getHBaseConfiguration URL: https://github.com/apache/flink/pull/22559 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721999#comment-17721999
]
Lijie Wang edited comment on FLINK-31706 at 5/12/23 3:59 AM:
-
Thanks for your attention and effort, but we cannot do it now, because it will
change the behavior of {{parallelism.default}} and
{{execution.batch.adaptive.auto-parallelism.default-source-parallelism}} ,
needs to be discussed and approved on the mailing list (See
[FLIP|https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals]
for details).
In addition, as I said above, we have some other ideas about Source paralleism
in adaptive batch scheduler, and hope to merge the idea here into one FLIP and
discuss them together on the mailing list.
This FLIP is in preparation and we will drive a discussion on the mailing list
once it is ready. If you are interested, welcome to participate in the
discussion and coding at that time.
WDYT? [~zxcoccer]
was (Author: wanglijie95):
Thanks for your attention and effort, but we cannot do it now, because it will
change the behavior of {{parallelism.default}} and
{{execution.batch.adaptive.auto-parallelism.default-source-parallelism}} ,
needs to be discussed and approved on the mailing list (See
[FLIP|https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals]
for details).
In addition, as I said above, we have some other ideas about Source paralleism
in adaptive batch scheduler, and hope to merge the idea here into one FLIP and
discuss them together on the mailing list.
This FLIP is in preparation and we will drive a discussion on the mailing list
once it is ready. If you are interested, welcome to participate in the
discussion and development at that time.
WDYT? [~zxcoccer]
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721999#comment-17721999
]
Lijie Wang commented on FLINK-31706:
Thanks for your attention and effort, but we cannot do it now, because it will
change the behavior of {{parallelism.default}} and
{{execution.batch.adaptive.auto-parallelism.default-source-parallelism}} ,
needs to be discussed and approved on the mailing list (See
[FLIP|https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals]
for details).
In addition, as I said above, we have some other ideas about Source paralleism
in adaptive batch scheduler, and hope to merge the idea here into one FLIP and
discuss them together on the mailing list.
This FLIP is in preparation and we will drive a discussion on the mailing list
once it is ready. If you are interested, welcome to participate in the
discussion and development at that time.
WDYT? [~zxcoccer]
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] xintongsong commented on a diff in pull request #22352: [FLINK-31639][network] Introduce tiered store memory manager
xintongsong commented on code in PR #22352:
URL: https://github.com/apache/flink/pull/22352#discussion_r1191854430
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImpl.java:
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.network.buffer.BufferBuilder;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.LocalBufferPool;
+import org.apache.flink.util.ExceptionUtils;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The implementation for {@link TieredStorageMemoryManager}. This is to
request or recycle buffers
+ * from {@link LocalBufferPool} for different memory owners, for example, the
tiers, the buffer
+ * accumulator, etc.
+ *
+ * Note that the memory owner should register its {@link
TieredStorageMemorySpec} firstly before
+ * requesting buffers.
+ */
+public class TieredStorageMemoryManagerImpl implements
TieredStorageMemoryManager {
+
+private final Map tieredMemorySpecs;
+
+private int numTotalExclusiveBuffers;
+
+private BufferPool bufferPool;
+
+private final AtomicInteger numRequestedBuffers;
+
+public TieredStorageMemoryManagerImpl() {
+this.tieredMemorySpecs = new HashMap<>();
+this.numRequestedBuffers = new AtomicInteger(0);
+}
+
+@Override
+public void setup(BufferPool bufferPool) {
+this.bufferPool = bufferPool;
+}
+
+@Override
+public void registerMemorySpec(TieredStorageMemorySpec memorySpec) {
+checkState(
+!tieredMemorySpecs.containsKey(memorySpec.getOwner()),
+"Duplicated memory spec registration.");
+tieredMemorySpecs.put(memorySpec.getOwner(), memorySpec);
+numTotalExclusiveBuffers += memorySpec.getNumExclusiveBuffers();
+}
+
+@Override
+public BufferBuilder requestBufferBlocking() {
+MemorySegment requestedBuffer = null;
+try {
+requestedBuffer = bufferPool.requestMemorySegmentBlocking();
+} catch (Throwable throwable) {
+ExceptionUtils.rethrow(throwable, "Failed to request memory
segments.");
+}
+numRequestedBuffers.incrementAndGet();
+return new BufferBuilder(checkNotNull(requestedBuffer),
this::recycleBuffer);
+}
Review Comment:
We are not checking whether a buffer can be requested. This should be
explained.
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/CacheFlushManager.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage;
+
+import
org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+
+import
org.apache.flink.shaded.guava30.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import
[jira] [Commented] (FLINK-32053) Introduce StateMetadata to ExecNode to support configure operator-level state TTL via CompiledPlan
[
https://issues.apache.org/jira/browse/FLINK-32053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721994#comment-17721994
]
Jane Chan commented on FLINK-32053:
---
Wait for FLINK-32064 to be merged
> Introduce StateMetadata to ExecNode to support configure operator-level state
> TTL via CompiledPlan
> --
>
> Key: FLINK-32053
> URL: https://issues.apache.org/jira/browse/FLINK-32053
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: Jane Chan
>Assignee: Jane Chan
>Priority: Major
> Fix For: 1.18.0
>
>
> This subtask should introduce StateMetadata to all ExecNodes that translate
> to stateful operators, changing the way how `#translateToPlanInternal` get
> the state retention time. The affected `ExecNode` list
> {code:java}
> StreamExecChangelogNormalize
> StreamExecDeduplicate
> StreamExecGlobalGroupAggregate
> StreamExecGroupAggregate
> StreamExecIncrementalGroupAggregate
> StreamExecJoin
> StreamExecLimit
> StreamExecLookupJoin
> StreamExecRank
> StreamExecSink
> StreamExecSortLimit
> {code}
> Since we have upgraded some `ExecNode`s to version 2, we have to test the
> following 3 parts:
> 1. the plans serialized using version 1 can be deserialized using the current
> version.
> 2. the plans with the current version SerDe work as expected.
> 3. The way by modifying the JSON content to change state TTL works as
> expected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-32064) Add subdirectory of test output file for JsonPlanTest to indicate the plan's version
[ https://issues.apache.org/jira/browse/FLINK-32064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoyuxia reassigned FLINK-32064: Assignee: Jane Chan > Add subdirectory of test output file for JsonPlanTest to indicate the plan's > version > - > > Key: FLINK-32064 > URL: https://issues.apache.org/jira/browse/FLINK-32064 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Jane Chan >Assignee: Jane Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > > This is a preparation for upgrading some ExecNodes(which translate to > stateful operators) version to 2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-31876) [JUnit5 Migration] Module: flink-queryable-state
[ https://issues.apache.org/jira/browse/FLINK-31876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo resolved FLINK-31876. Fix Version/s: 1.18.0 Resolution: Done > [JUnit5 Migration] Module: flink-queryable-state > > > Key: FLINK-31876 > URL: https://issues.apache.org/jira/browse/FLINK-31876 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Queryable State >Affects Versions: 1.18.0 >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Minor > Labels: pull-request-available > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31876) [JUnit5 Migration] Module: flink-queryable-state
[ https://issues.apache.org/jira/browse/FLINK-31876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17716954#comment-17716954 ] Weijie Guo edited comment on FLINK-31876 at 5/12/23 3:05 AM: - master(1.18) migrate flink-queryable-state-client via 8be25d15501744f809e31208757ff211558710ae. migrate flink-queryable-state-runtime via 7f4e834babe7e8865123148e57ecdd6a138b609d. was (Author: weijie guo): migrate flink-queryable-state-client via 8be25d15501744f809e31208757ff211558710ae. > [JUnit5 Migration] Module: flink-queryable-state > > > Key: FLINK-31876 > URL: https://issues.apache.org/jira/browse/FLINK-31876 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Queryable State >Affects Versions: 1.18.0 >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29618) YARNSessionFIFOSecuredITCase.testDetachedMode timed out in Azure CI
[
https://issues.apache.org/jira/browse/FLINK-29618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-29618:
---
Labels: pull-request-available starter test-stability (was: starter
test-stability)
> YARNSessionFIFOSecuredITCase.testDetachedMode timed out in Azure CI
> ---
>
> Key: FLINK-29618
> URL: https://issues.apache.org/jira/browse/FLINK-29618
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, starter, test-stability
> Attachments:
> build-20221012.7.YARNSessionFIFOSecuredITCase.testDetachedMode.log
>
>
> We experienced a [build
> failure|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41931=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=30284]
> that was caused (exclusively) by
> {{YARNSessionFIFOSecuredITCase.testDetachedMode}} running into a timeout.
> The test specific logs which were extracted from the build's are attached to
> this Jira issue.
> JUnit tries to stop the thread running the test but fails to due so because
> it's interrupting a sleep. The {{InterruptedException}} is not properly
> handled in
> [YarnTestBase:744|https://github.com/apache/flink/blob/573ed922346c791760d27653543c2b8df56f51f7/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java#L744]
> (it doesn't forward the exception). Therefore, we only see the warning being
> logged after 60s:
> {code}
> 11:33:51,124 [ForkJoinPool-1-worker-25] WARN
> org.apache.flink.yarn.YarnTestBase [] - Interruped
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method) ~[?:1.8.0_292]
> at org.apache.flink.yarn.YarnTestBase.sleep(YarnTestBase.java:716)
> ~[test-classes/:?]
> at
> org.apache.flink.yarn.YarnTestBase.startWithArgs(YarnTestBase.java:906)
> ~[test-classes/:?]
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.runDetachedModeTest(YARNSessionFIFOITCase.java:141)
> ~[test-classes/:?]
> at
> org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.lambda$testDetachedMode$2(YARNSessionFIFOSecuredITCase.java:173)
> ~[test-classes/:?]
> at org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:288)
> ~[test-classes/:?]
> at
> org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.testDetachedMode(YARNSessionFIFOSecuredITCase.java:160)
> ~[test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_292]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_292]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_292]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
> [...]
> {code}
> The test code itself eventually continues and succeeds (despite the
> interruption). The job submission takes suspiciously long, though.
> Removing the timeout from the test (as this is the desired approach for tests
> in general now) should solve this test instability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] fredia commented on pull request #22457: [FLINK-31876][QS] Migrate flink-queryable-state-runtime tests to JUnit5
fredia commented on PR #22457: URL: https://github.com/apache/flink/pull/22457#issuecomment-1545030267 @reswqa CI turns green, but the result is not synchronized to flinkbot, could you help merge it? thanks. https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48819=results -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #22571: [FLINK-32056][Python][Connector/Pulsar] Upgrade flink-connector-pulsar in flink-python to v4.0.0
dianfu commented on PR #22571: URL: https://github.com/apache/flink/pull/22571#issuecomment-1545020164 I noticed that the documentation of 1.17 has already been updated to 4.0.0-1.17, see more details on https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/pulsar/ I guess we need also port this to release-1.17? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #21971: [FLINK-31084][connectors/dataGen] Add default value for dataGen seque…
xuzhiwen1255 commented on code in PR #21971:
URL: https://github.com/apache/flink/pull/21971#discussion_r1191834141
##
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/DataGenTableSource.java:
##
@@ -93,4 +93,9 @@ public ChangelogMode getChangelogMode() {
public void applyLimit(long limit) {
this.numberOfRows = limit;
}
+
+@VisibleForTesting
+public DataGenerator[] getFieldGenerators() {
Review Comment:
Yes, but should we change the package structure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31967) SQL with LAG function NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721969#comment-17721969
]
Shengkai Fang commented on FLINK-31967:
---
[~padavan] Sure. There will be a PR to fix this soon.
> SQL with LAG function NullPointerException
> --
>
> Key: FLINK-31967
> URL: https://issues.apache.org/jira/browse/FLINK-31967
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: padavan
>Priority: Major
> Attachments: image-2023-04-28-14-46-19-736.png,
> image-2023-04-28-15-06-48-184.png, image-2023-04-28-15-14-58-788.png,
> image-2023-04-28-15-17-49-144.png, image-2023-04-28-17-06-20-737.png,
> simpleFlinkKafkaLag.zip
>
>
> I want to make a query with the LAG function. And got Job Exception without
> any explanations.
>
> *Code:*
> {code:java}
> private static void t1_LeadLag(DataStream ds,
> StreamExecutionEnvironment env) {
> StreamTableEnvironment te = StreamTableEnvironment.create(env);
> Table t = te.fromDataStream(ds,
> Schema.newBuilder().columnByExpression("proctime", "proctime()").build());
> te.createTemporaryView("users", t);
> Table res = te.sqlQuery("SELECT userId, `count`,\n" +
> " LAG(`count`) OVER (PARTITION BY userId ORDER BY proctime) AS
> prev_quantity\n" +
> " FROM users");
> te.toChangelogStream(res).print();
> }{code}
>
> *Input:*
> {"userId":3,"count":0,"dt":"2023-04-28T07:44:21.551Z"}
>
> *Exception:* I remove part about basic JobExecutionException and kept the
> important(i think)
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.table.data.GenericRowData.getInt(GenericRowData.java:149)
> at
> org.apache.flink.table.data.RowData.lambda$createFieldGetter$245ca7d1$6(RowData.java:245)
> at
> org$apache$flink$table$runtime$functions$aggregate$LagAggFunction$LagAcc$2$Converter.toExternal(Unknown
> Source)
> at
> org.apache.flink.table.data.conversion.StructuredObjectConverter.toExternal(StructuredObjectConverter.java:101)
> at UnboundedOverAggregateHelper$15.setAccumulators(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.over.ProcTimeUnboundedPrecedingFunction.processElement(ProcTimeUnboundedPrecedingFunction.java:92)
> at
> org.apache.flink.table.runtime.operators.over.ProcTimeUnboundedPrecedingFunction.processElement(ProcTimeUnboundedPrecedingFunction.java:42)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Thread.java:829){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:26 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[pr-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments? thanks a lot.
[~wanglijie]
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[pr-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments? thanks a lot.
[~wanglijie]
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:26 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[pr-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments? thanks a lot.
[~wanglijie]
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[pr|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
[link
title|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:24 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[pr|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
[link
title|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[pr|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:24 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[pr|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[PR-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:24 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[PR-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[PR-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
[link title|http://example.com]
can you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:23 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[PR-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
[link title|http://example.com]
can you spare some precious time to review and make some comments.
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[PR-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]can
you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:22 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in
[PR-22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]can
you spare some precious time to review and make some comments.
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in [PR
22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]can
you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou edited comment on FLINK-31706 at 5/12/23 12:22 AM:
-
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in [PR
22555|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]can
you spare some precious time to review and make some comments.
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in [[PR
22555]|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721957#comment-17721957
]
xiaochen zhou commented on FLINK-31706:
---
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in [[PR
22555]|[https://github.com/apache/flink/pull/22555|https://github.com/apache/flink/pull/22555)]]
can you spare some precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[ https://issues.apache.org/jira/browse/FLINK-31706 ]
xiaochen zhou deleted comment on FLINK-31706:
---
was (Author: zxcoccer):
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in [pr
22555]([https://github.com/apache/flink/pull/22555)] can you spare some
precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721956#comment-17721956
]
xiaochen zhou commented on FLINK-31706:
---
I am currently implementing the use `paralleism.default` instread of the
`execution.batch.adaptive.auto-parallelism.default-source-parallelism.` in [pr
22555]([https://github.com/apache/flink/pull/22555)] can you spare some
precious time to review and make some comments.
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31967) SQL with LAG function NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721936#comment-17721936
]
padavan commented on FLINK-31967:
-
[~fsk119] with Integer work.
I think it is at least unexpected behavior that int != Integer. Will there be
any fixes? Or will you just improve the output error with a description?
> SQL with LAG function NullPointerException
> --
>
> Key: FLINK-31967
> URL: https://issues.apache.org/jira/browse/FLINK-31967
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: padavan
>Priority: Major
> Attachments: image-2023-04-28-14-46-19-736.png,
> image-2023-04-28-15-06-48-184.png, image-2023-04-28-15-14-58-788.png,
> image-2023-04-28-15-17-49-144.png, image-2023-04-28-17-06-20-737.png,
> simpleFlinkKafkaLag.zip
>
>
> I want to make a query with the LAG function. And got Job Exception without
> any explanations.
>
> *Code:*
> {code:java}
> private static void t1_LeadLag(DataStream ds,
> StreamExecutionEnvironment env) {
> StreamTableEnvironment te = StreamTableEnvironment.create(env);
> Table t = te.fromDataStream(ds,
> Schema.newBuilder().columnByExpression("proctime", "proctime()").build());
> te.createTemporaryView("users", t);
> Table res = te.sqlQuery("SELECT userId, `count`,\n" +
> " LAG(`count`) OVER (PARTITION BY userId ORDER BY proctime) AS
> prev_quantity\n" +
> " FROM users");
> te.toChangelogStream(res).print();
> }{code}
>
> *Input:*
> {"userId":3,"count":0,"dt":"2023-04-28T07:44:21.551Z"}
>
> *Exception:* I remove part about basic JobExecutionException and kept the
> important(i think)
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.table.data.GenericRowData.getInt(GenericRowData.java:149)
> at
> org.apache.flink.table.data.RowData.lambda$createFieldGetter$245ca7d1$6(RowData.java:245)
> at
> org$apache$flink$table$runtime$functions$aggregate$LagAggFunction$LagAcc$2$Converter.toExternal(Unknown
> Source)
> at
> org.apache.flink.table.data.conversion.StructuredObjectConverter.toExternal(StructuredObjectConverter.java:101)
> at UnboundedOverAggregateHelper$15.setAccumulators(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.over.ProcTimeUnboundedPrecedingFunction.processElement(ProcTimeUnboundedPrecedingFunction.java:92)
> at
> org.apache.flink.table.runtime.operators.over.ProcTimeUnboundedPrecedingFunction.processElement(ProcTimeUnboundedPrecedingFunction.java:42)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Thread.java:829){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol commented on a diff in pull request #21349: [FLINK-28203] Support Maven 3.3+
zentol commented on code in PR #21349:
URL: https://github.com/apache/flink/pull/21349#discussion_r1191632889
##
tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/optional/ShadeOptionalChecker.java:
##
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tools.ci.optional;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.tools.ci.utils.dependency.DependencyParser;
+import org.apache.flink.tools.ci.utils.shade.ShadeParser;
+import org.apache.flink.tools.ci.utils.shared.Dependency;
+import org.apache.flink.tools.ci.utils.shared.DependencyTree;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Verifies that all dependencies bundled with the shade-plugin are marked as
optional in the pom.
+ * This ensures compatibility with later maven versions and in general
simplifies dependency
+ * management as transitivity is no longer dependent on the shade-plugin.
+ *
+ * In Maven 3.3 the dependency tree was made immutable at runtime, and thus
can no longer be
+ * changed by the shade plugin. The plugin would usually remove a dependency
from the tree when it
+ * is being bundled (known as dependency reduction). While dependency
reduction still works for the
+ * published poms (== what users consume) since it can still change the
content of the final pom,
+ * while developing Flink it no longer works. This breaks plenty of things,
since suddenly a bunch
+ * of dependencies are still visible to downstream modules that weren't before.
+ *
+ * To workaround this we mark all dependencies that we bundle as optional;
this makes them
+ * non-transitive. To a downstream module, behavior-wise a non-transitive
dependency is identical to
+ * a removed dependency.
+ *
+ * This checker analyzes the bundled dependencies (based on the
shade-plugin output) and the set
+ * of dependencies (based on the dependency plugin) to detect cases where a
dependency is not marked
+ * as optional as it should.
+ *
+ * The enforced rule is rather simple: Any dependency that is bundled, or
any of its parents,
+ * must show up as optional in the dependency tree. The parent clause is
required to cover cases
+ * where a module has 2 paths to a bundled dependency. If a module depends on
A1/A2, each depending
+ * on B, with A1 and B being bundled, then even if A1 is marked as optional B
is still shown as a
+ * non-optional dependency (because the non-optional A2 still needs it!).
+ */
+public class ShadeOptionalChecker {
+private static final Logger LOG =
LoggerFactory.getLogger(ShadeOptionalChecker.class);
+
+public static void main(String[] args) throws IOException {
+if (args.length < 2) {
+System.out.println(
+"Usage: ShadeOptionalChecker
");
+System.exit(1);
+}
+
+final Path shadeOutputPath = Paths.get(args[0]);
+final Path dependencyOutputPath = Paths.get(args[1]);
+
+final Map> bundledDependenciesByModule =
+ShadeParser.parseShadeOutput(shadeOutputPath);
+final Map dependenciesByModule =
+
DependencyParser.parseDependencyTreeOutput(dependencyOutputPath);
+
+final Map> violations =
+checkOptionalFlags(bundledDependenciesByModule,
dependenciesByModule);
+
+if (!violations.isEmpty()) {
+LOG.error(
+"{} modules bundle in total {} dependencies without them
being marked as optional in the pom.",
+violations.keySet().size(),
+violations.size());
Review Comment:
This is annoying. I actually noticed that on CI but forgot to fix it :cry:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[jira] [Closed] (FLINK-31934) Remove mocking in RocksDB tests
[ https://issues.apache.org/jira/browse/FLINK-31934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-31934. Resolution: Fixed master: 2d873629b43ec7339850b685833e3404d5ae824a > Remove mocking in RocksDB tests > --- > > Key: FLINK-31934 > URL: https://issues.apache.org/jira/browse/FLINK-31934 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends, Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol merged pull request #22484: [FLINK-31934][rocksdb][tests] Remove mocking
zentol merged PR #22484: URL: https://github.com/apache/flink/pull/22484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-31913) sql-client.sh does not respect env.java.opts.all/client
[ https://issues.apache.org/jira/browse/FLINK-31913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-31913. Resolution: Fixed master: 49a5810f0eafad277f81c652435842e65b632169 > sql-client.sh does not respect env.java.opts.all/client > --- > > Key: FLINK-31913 > URL: https://issues.apache.org/jira/browse/FLINK-31913 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Scripts, Table SQL / Client >Affects Versions: 1.17.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol merged pull request #22475: [FLINK-31913][sql] SQL client respects env.java.opts.client
zentol merged PR #22475: URL: https://github.com/apache/flink/pull/22475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22571: [FLINK-32056][Python][Connector/Pulsar] Upgrade flink-connector-pulsar in flink-python to v4.0.0
flinkbot commented on PR #22571: URL: https://github.com/apache/flink/pull/22571#issuecomment-1544498302 ## CI report: * e142db78820c242c71c56fafecedc9e9bdadb7f7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-32056) Update the used Pulsar connector in flink-python to 4.0.0
[ https://issues.apache.org/jira/browse/FLINK-32056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32056: --- Labels: pull-request-available (was: ) > Update the used Pulsar connector in flink-python to 4.0.0 > - > > Key: FLINK-32056 > URL: https://issues.apache.org/jira/browse/FLINK-32056 > Project: Flink > Issue Type: Bug > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.18.0, 1.17.1 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > > flink-python still references and tests flink-connector-pulsar:3.0.0, while > it should be using flink-connector-pulsar:4.0.0. That's because the newer > version is the only version compatible with Flink 1.17 and it doesn't rely on > flink-shaded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] MartijnVisser opened a new pull request, #22571: [FLINK-32056][Python][Connector/Pulsar] Upgrade flink-connector-pulsar in flink-python to v4.0.0
MartijnVisser opened a new pull request, #22571: URL: https://github.com/apache/flink/pull/22571 ## What is the purpose of the change * Upgrade used `flink-connector-pulsar` in `flink-python` to 4.0.0, since that's compatible with Flink 1.17+. Without this upgrade, we can't upgrade `flink-shaded` since the older versions of `flink-connector-pulsar` directly use `flink-shaded` and there's a breaking change in `flink-shaded`. ## Brief change log * Upgrade to `flink-connector-pulsar` to 4.0.0 * Remove and refactoring all occurrences of `PulsarSerializationSchema` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-mongodb] boring-cyborg[bot] commented on pull request #7: [FLINK-32024][docs] Short code related to externalized connector retr…
boring-cyborg[bot] commented on PR #7: URL: https://github.com/apache/flink-connector-mongodb/pull/7#issuecomment-1544433289 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-aws] reswqa opened a new pull request, #74: [FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #74: URL: https://github.com/apache/flink-connector-aws/pull/74 Backport FLINK-32024 to `v4.1` branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-aws] reswqa opened a new pull request, #73: [FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #73: URL: https://github.com/apache/flink-connector-aws/pull/73 ## Purpose of the change * Short code related to externalized connector retrieve version from its own data yaml.* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Significant changes - [ ] Dependencies have been added or upgraded - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [ ] New feature has been introduced - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-rabbitmq] reswqa opened a new pull request, #13: [BP-3.0][FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #13: URL: https://github.com/apache/flink-connector-rabbitmq/pull/13 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-rabbitmq] reswqa opened a new pull request, #12: [FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #12: URL: https://github.com/apache/flink-connector-rabbitmq/pull/12 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29533) Add proper table style to Flink website
[ https://issues.apache.org/jira/browse/FLINK-29533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721853#comment-17721853 ] Sakshi Sharma commented on FLINK-29533: --- Thank you so much. Will definitely take a look at the example and start working on the issue. > Add proper table style to Flink website > --- > > Key: FLINK-29533 > URL: https://issues.apache.org/jira/browse/FLINK-29533 > Project: Flink > Issue Type: Bug > Components: Project Website >Affects Versions: 1.16.0, 1.17.0 >Reporter: Matthias Pohl >Assignee: Sakshi Sharma >Priority: Major > Labels: starter > Attachments: Screenshot from 2022-10-07 08-23-01.png > > > Tables can be created using simple markdown syntax. But the corresponding > rendered table lacks proper styling: !Screenshot from 2022-10-07 > 08-23-01.png! > Several blog post work around that by adding a custom style: > * [Apache Flink Kubernetes Operator 1.0.0 Release > Announcement|https://flink.apache.org/news/2022/06/05/release-kubernetes-operator-1.0.0.html] > * [Improving speed and stability of checkpointing with generic log-based > incremental > checkpoints|https://flink.apache.org/2022/05/30/changelog-state-backend.html] > What about coming up with a common style that doesn't require people to come > up with their own custom style per post. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-pulsar] reswqa opened a new pull request, #49: [BP-3.0][FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #49: URL: https://github.com/apache/flink-connector-pulsar/pull/49 Backport FLINK-32024 to `v3.0` branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-pulsar] reswqa opened a new pull request, #48: [FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #48: URL: https://github.com/apache/flink-connector-pulsar/pull/48 ## Purpose of the change *As Title* ## Brief change log - *Remove the version parameter from `shortcode`.* - *Add version info to `pulsar.yml`.* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Significant changes *(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)* - [ ] Dependencies have been added or upgraded - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [ ] New feature has been introduced - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-aws] hlteoh37 commented on a diff in pull request #49: [FLINK-24438] Add Kinesis connector using FLIP-27
hlteoh37 commented on code in PR #49:
URL:
https://github.com/apache/flink-connector-aws/pull/49#discussion_r1191432157
##
flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/source/enumerator/assigner/HashShardAssigner.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kinesis.source.enumerator.assigner;
+
+import org.apache.flink.annotation.Experimental;
+import
org.apache.flink.connector.kinesis.source.enumerator.KinesisShardAssigner;
+import org.apache.flink.connector.kinesis.source.split.KinesisShardSplit;
+import org.apache.flink.util.Preconditions;
+
+/** An implementation of the {@link KinesisShardAssigner} that assigns splits
by hashcode. */
+@Experimental
Review Comment:
Good suggestion. Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-connector-jdbc] reswqa opened a new pull request, #47: [FLINK-32024][docs] Short code related to externalized connector retr…
reswqa opened a new pull request, #47: URL: https://github.com/apache/flink-connector-jdbc/pull/47 …ieve version from its own data yaml -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-jdbc] reswqa opened a new pull request, #48: [BP-3.0][FLINK-32024][docs] Short code related to externalized connector retr…
reswqa opened a new pull request, #48: URL: https://github.com/apache/flink-connector-jdbc/pull/48 …ieve version from its own data yaml -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-32014) Cassandra source documentation is missing and javadoc is out of sync
[ https://issues.apache.org/jira/browse/FLINK-32014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot resolved FLINK-32014. -- Fix Version/s: 1.18.0 Resolution: Fixed > Cassandra source documentation is missing and javadoc is out of sync > > > Key: FLINK-32014 > URL: https://issues.apache.org/jira/browse/FLINK-32014 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Cassandra, Documentation >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-hbase] reswqa opened a new pull request, #11: [BP-3.0][FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #11: URL: https://github.com/apache/flink-connector-hbase/pull/11 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-hbase] reswqa opened a new pull request, #10: [FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #10: URL: https://github.com/apache/flink-connector-hbase/pull/10 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32047) Fix args in JobSpec not being passed through to Flink in Standalone mode - 1.4.0
[
https://issues.apache.org/jira/browse/FLINK-32047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721834#comment-17721834
]
Usamah Jassat commented on FLINK-32047:
---
I can look into it
> Fix args in JobSpec not being passed through to Flink in Standalone mode -
> 1.4.0
>
>
> Key: FLINK-32047
> URL: https://issues.apache.org/jira/browse/FLINK-32047
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gil Shmaya
>Priority: Major
> Attachments: image-2023-04-30-18-54-22-291.png,
> image-2023-04-30-19-56-30-150.png, image-2023-04-30-19-56-57-680.png
>
>
> This issue is related to a previously fixed bug in version 1.2.0 -
> FLINK-29388
> I have noticed that while the args are successfully being passed when using
> version 1.2.0, this is not the case with version 1.4.0.
> {+}Scenario{+}:
> I added a log that prints the argument array length at the beginning of the
> main function of the flink job:
> !image-2023-04-30-18-54-22-291.png!
> The result when running with 1.2.0:
> !image-2023-04-30-19-56-30-150.png!
> The result when running with 1.4.0:
> !image-2023-04-30-19-56-57-680.png!
> h4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-connector-cassandra] echauchot merged pull request #15: [FLINK-32014][doc][hotfix] Add Cassandra source documentation to website
echauchot merged PR #15: URL: https://github.com/apache/flink-connector-cassandra/pull/15 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1191359995
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##
@@ -65,33 +72,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been
initialized.");
-this.checkpointedState =
-context.getOperatorStateStore()
-.getListState(
-new ListStateDescriptor<>(
-name + "-sequence-state",
LongSerializer.INSTANCE));
-this.valuesToEmit = new ArrayDeque<>();
-if (context.isRestored()) {
-// upon restoring
+ListStateDescriptor stateDescriptor =
+new ListStateDescriptor<>(
+name + "-sequence-state",
TypeInformation.of(InternalState.class));
+this.checkpointedState =
context.getOperatorStateStore().getListState(stateDescriptor);
+this.internalStates = Lists.newArrayList();
-for (Long v : this.checkpointedState.get()) {
-this.valuesToEmit.add(v);
-}
+totalNoOfElements = Math.abs(end - start + 1);
+if (context.isRestored()) {
+checkpointedState.get().forEach(state ->
internalStates.add(state));
} else {
// the first time the job is executed
-final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
-final long congruence = start + taskIdx;
+final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+internalStates.add(new InternalState(0, taskIdx, stepSize));
+}
+}
-long totalNoOfElements = Math.abs(end - start + 1);
-final int baseSize = safeDivide(totalNoOfElements, stepSize);
-final int toCollect =
-(totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 :
baseSize;
+private long toCollect(long baseSize, long stepSize, int taskIdx) {
+return (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 :
baseSize;
+}
-for (long collected = 0; collected < toCollect; collected++) {
-this.valuesToEmit.add(collected * stepSize + congruence);
+public Long nextValue() {
+Iterator iterator = internalStates.iterator();
+if (iterator.hasNext()) {
+InternalState state = iterator.next();
+long nextSequence = state.collected * state.stepSize + (start +
state.taskId);
+state.collected++;
Review Comment:
If we are updating the current state at every iteration anyway: instead of
incrementing `collected`, why not just retain the value in the current state?
It's easy to calculate the *next* (just add stepSize) and you know to drop the
state from the list if it goes over `this.end`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-connector-gcp-pubsub] reswqa opened a new pull request, #9: [BP-3.0][FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #9: URL: https://github.com/apache/flink-connector-gcp-pubsub/pull/9 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-gcp-pubsub] reswqa opened a new pull request, #8: [FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml
reswqa opened a new pull request, #8: URL: https://github.com/apache/flink-connector-gcp-pubsub/pull/8 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LadyForest commented on a diff in pull request #22539: [FLINK-31956][table] Extend the CompiledPlan to read from/write to Fl…
LadyForest commented on code in PR #22539:
URL: https://github.com/apache/flink/pull/22539#discussion_r1191301752
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/CompiledPlan.java:
##
@@ -109,6 +109,8 @@ default void writeToFile(File file) {
*/
void writeToFile(File file, boolean ignoreIfExists);
+void writeToFile(org.apache.flink.core.fs.Path path, boolean
ignoreIfExists);
Review Comment:
I think there's no need to add an extra interface method to achieve the goal.
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/delegation/InternalPlan.java:
##
@@ -44,6 +45,13 @@ public interface InternalPlan {
*/
void writeToFile(File file, boolean ignoreIfExists, boolean failIfExists);
+/**
+ * Note that {@code ignoreIfExists} has precedence over {@code
failIfExists}.
+ *
+ * @see CompiledPlan#writeToFile(Path, boolean)
+ */
+void writeToFile(Path path, boolean ignoreIfExists, boolean failIfExists);
Review Comment:
ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #21349: [FLINK-28203] Support Maven 3.3+
XComp commented on code in PR #21349:
URL: https://github.com/apache/flink/pull/21349#discussion_r1191307099
##
tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/optional/ShadeOptionalChecker.java:
##
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tools.ci.optional;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.tools.ci.utils.dependency.DependencyParser;
+import org.apache.flink.tools.ci.utils.shade.ShadeParser;
+import org.apache.flink.tools.ci.utils.shared.Dependency;
+import org.apache.flink.tools.ci.utils.shared.DependencyTree;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Verifies that all dependencies bundled with the shade-plugin are marked as
optional in the pom.
+ * This ensures compatibility with later maven versions and in general
simplifies dependency
+ * management as transitivity is no longer dependent on the shade-plugin.
+ *
+ * In Maven 3.3 the dependency tree was made immutable at runtime, and thus
can no longer be
+ * changed by the shade plugin. The plugin would usually remove a dependency
from the tree when it
+ * is being bundled (known as dependency reduction). While dependency
reduction still works for the
+ * published poms (== what users consume) since it can still change the
content of the final pom,
+ * while developing Flink it no longer works. This breaks plenty of things,
since suddenly a bunch
+ * of dependencies are still visible to downstream modules that weren't before.
+ *
+ * To workaround this we mark all dependencies that we bundle as optional;
this makes them
+ * non-transitive. To a downstream module, behavior-wise a non-transitive
dependency is identical to
+ * a removed dependency.
+ *
+ * This checker analyzes the bundled dependencies (based on the
shade-plugin output) and the set
+ * of dependencies (based on the dependency plugin) to detect cases where a
dependency is not marked
+ * as optional as it should.
+ *
+ * The enforced rule is rather simple: Any dependency that is bundled, or
any of its parents,
+ * must show up as optional in the dependency tree. The parent clause is
required to cover cases
+ * where a module has 2 paths to a bundled dependency. If a module depends on
A1/A2, each depending
+ * on B, with A1 and B being bundled, then even if A1 is marked as optional B
is still shown as a
+ * non-optional dependency (because the non-optional A2 still needs it!).
+ */
+public class ShadeOptionalChecker {
+private static final Logger LOG =
LoggerFactory.getLogger(ShadeOptionalChecker.class);
+
+public static void main(String[] args) throws IOException {
+if (args.length < 2) {
+System.out.println(
+"Usage: ShadeOptionalChecker
");
+System.exit(1);
+}
+
+final Path shadeOutputPath = Paths.get(args[0]);
+final Path dependencyOutputPath = Paths.get(args[1]);
+
+final Map> bundledDependenciesByModule =
+ShadeParser.parseShadeOutput(shadeOutputPath);
+final Map dependenciesByModule =
+
DependencyParser.parseDependencyTreeOutput(dependencyOutputPath);
+
+final Map> violations =
+checkOptionalFlags(bundledDependenciesByModule,
dependenciesByModule);
+
+if (!violations.isEmpty()) {
+LOG.error(
+"{} modules bundle in total {} dependencies without them
being marked as optional in the pom.",
+violations.keySet().size(),
+violations.size());
Review Comment:
```suggestion
violations.values().stream().mapToInt(Set::size).sum());
```
Not sure why I missed it. But other than that, the description looks
alright.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
[GitHub] [flink] flinkbot commented on pull request #22570: [FLINK-32064][table-planner] Add subdirectory of test output file for JsonPlanTest to indicate the plan's version
flinkbot commented on PR #22570: URL: https://github.com/apache/flink/pull/22570#issuecomment-1544129227 ## CI report: * dc25ba1c1d35f84b0164615d022e997ca94d1d03 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-32064) Add subdirectory of test output file for JsonPlanTest to indicate the plan's version
[ https://issues.apache.org/jira/browse/FLINK-32064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32064: --- Labels: pull-request-available (was: ) > Add subdirectory of test output file for JsonPlanTest to indicate the plan's > version > - > > Key: FLINK-32064 > URL: https://issues.apache.org/jira/browse/FLINK-32064 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Jane Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > > This is a preparation for upgrading some ExecNodes(which translate to > stateful operators) version to 2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] LadyForest opened a new pull request, #22570: [FLINK-32064][table-planner] Add subdirectory of test output file for JsonPlanTest to indicate the plan's version
LadyForest opened a new pull request, #22570: URL: https://github.com/apache/flink/pull/22570 ## What is the purpose of the change This PR is a refactor for all JsonPlanTest, as preparation for upgrading some ExecNodes to version 2. ## Brief change log - Add a subdirectory `execNodeVersion=1` under each JsonPlanTest's output file. - Refactor the `verifyJsonPlan` method, to walk through all subdirectories under each PlanTest, check JsonSerDe for each version, and use the highest version to check object <--> JSON works as expected. ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduces a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32038) OffsetCommitMode.Kafka_periodic with checkpointing enabled
[
https://issues.apache.org/jira/browse/FLINK-32038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721809#comment-17721809
]
Martijn Visser commented on FLINK-32038:
Changing default behavior would be a breaking change for Flink 1.x: I don't
think that is something that we should consider. I would still be included to
mark this as a Won't do. [~tzulitai] WDYT?
> OffsetCommitMode.Kafka_periodic with checkpointing enabled
> ---
>
> Key: FLINK-32038
> URL: https://issues.apache.org/jira/browse/FLINK-32038
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Runtime / Checkpointing
>Affects Versions: 1.14.6
>Reporter: Pritam Agarwala
>Priority: Major
>
> I need to get kafka-lag to prepare a graph and its dependent on kafka
> committed offset. Flink is updating the offsets only after checkpointing to
> make it consistent.
> Default Behaviour as per doc :
> If checkpoint is enabled, but {{consumer.setCommitOffsetsOnCheckpoints}} set
> to false, then offset will not be committed at all even if the
> {{enable.auto.commit}} is set to true.
> So, when {{consumer.setCommitOffsetsOnCheckpoints}} set to false, *shouldn't
> it fall back on the {{enable.auto.commit}} to do offset commit regularly
> since* *in any case flink doesn't use consumer committed offsets for
> recovery.*
>
> OffsetCommitModes class :
>
> {code:java}
> public class OffsetCommitModes {
> /**
> * Determine the offset commit mode using several configuration values.
> *
> * @param enableAutoCommit whether or not auto committing is enabled in
> the provided Kafka
> * properties.
> * @param enableCommitOnCheckpoint whether or not committing on
> checkpoints is enabled.
> * @param enableCheckpointing whether or not checkpoint is enabled for
> the consumer.
> * @return the offset commit mode to use, based on the configuration
> values.
> */
> public static OffsetCommitMode fromConfiguration(
> boolean enableAutoCommit,
> boolean enableCommitOnCheckpoint,
> boolean enableCheckpointing) {
> if (enableCheckpointing) {
> // if checkpointing is enabled, the mode depends only on whether
> committing on
> // checkpoints is enabled
> return (enableCommitOnCheckpoint)
> ? OffsetCommitMode.ON_CHECKPOINTS
> : OffsetCommitMode.DISABLED;
> } else {
> // else, the mode depends only on whether auto committing is
> enabled in the provided
> // Kafka properties
> return (enableAutoCommit) ? OffsetCommitMode.KAFKA_PERIODIC :
> OffsetCommitMode.DISABLED;
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32064) Add subdirectory of test output file for JsonPlanTest to indicate the plan's version
[ https://issues.apache.org/jira/browse/FLINK-32064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chan updated FLINK-32064: -- Summary: Add subdirectory of test output file for JsonPlanTest to indicate the plan's version (was: Add sub-directory of test output file for JsonPlanTest to indicate the plan's version) > Add subdirectory of test output file for JsonPlanTest to indicate the plan's > version > - > > Key: FLINK-32064 > URL: https://issues.apache.org/jira/browse/FLINK-32064 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Jane Chan >Priority: Major > Fix For: 1.18.0 > > > This is a preparation for upgrading some ExecNodes(which translate to > stateful operators) version to 2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32064) Add sub-directory of test output file for JsonPlanTest to indicate the plan's version
[ https://issues.apache.org/jira/browse/FLINK-32064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chan updated FLINK-32064: -- Description: This is a preparation for upgrading some ExecNodes(which translate to stateful operators) version to 2 (was: This is a preparation for upgrade ExecNode version to 2) > Add sub-directory of test output file for JsonPlanTest to indicate the > plan's version > -- > > Key: FLINK-32064 > URL: https://issues.apache.org/jira/browse/FLINK-32064 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Jane Chan >Priority: Major > Fix For: 1.18.0 > > > This is a preparation for upgrading some ExecNodes(which translate to > stateful operators) version to 2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32064) Add sub-directory of test output file for JsonPlanTest to indicate the plan's version
[ https://issues.apache.org/jira/browse/FLINK-32064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chan updated FLINK-32064: -- Description: This is a preparation for upgrade ExecNode version to 2 > Add sub-directory of test output file for JsonPlanTest to indicate the > plan's version > -- > > Key: FLINK-32064 > URL: https://issues.apache.org/jira/browse/FLINK-32064 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Jane Chan >Priority: Major > Fix For: 1.18.0 > > > This is a preparation for upgrade ExecNode version to 2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32064) Add sub-directory of test output file for JsonPlanTest to indicate the plan's version
Jane Chan created FLINK-32064: - Summary: Add sub-directory of test output file for JsonPlanTest to indicate the plan's version Key: FLINK-32064 URL: https://issues.apache.org/jira/browse/FLINK-32064 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.18.0 Reporter: Jane Chan Fix For: 1.18.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-32038) OffsetCommitMode.Kafka_periodic with checkpointing enabled
[
https://issues.apache.org/jira/browse/FLINK-32038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritam Agarwala reopened FLINK-32038:
-
> OffsetCommitMode.Kafka_periodic with checkpointing enabled
> ---
>
> Key: FLINK-32038
> URL: https://issues.apache.org/jira/browse/FLINK-32038
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Runtime / Checkpointing
>Affects Versions: 1.14.6
>Reporter: Pritam Agarwala
>Priority: Major
>
> I need to get kafka-lag to prepare a graph and its dependent on kafka
> committed offset. Flink is updating the offsets only after checkpointing to
> make it consistent.
> Default Behaviour as per doc :
> If checkpoint is enabled, but {{consumer.setCommitOffsetsOnCheckpoints}} set
> to false, then offset will not be committed at all even if the
> {{enable.auto.commit}} is set to true.
> So, when {{consumer.setCommitOffsetsOnCheckpoints}} set to false, *shouldn't
> it fall back on the {{enable.auto.commit}} to do offset commit regularly
> since* *in any case flink doesn't use consumer committed offsets for
> recovery.*
>
> OffsetCommitModes class :
>
> {code:java}
> public class OffsetCommitModes {
> /**
> * Determine the offset commit mode using several configuration values.
> *
> * @param enableAutoCommit whether or not auto committing is enabled in
> the provided Kafka
> * properties.
> * @param enableCommitOnCheckpoint whether or not committing on
> checkpoints is enabled.
> * @param enableCheckpointing whether or not checkpoint is enabled for
> the consumer.
> * @return the offset commit mode to use, based on the configuration
> values.
> */
> public static OffsetCommitMode fromConfiguration(
> boolean enableAutoCommit,
> boolean enableCommitOnCheckpoint,
> boolean enableCheckpointing) {
> if (enableCheckpointing) {
> // if checkpointing is enabled, the mode depends only on whether
> committing on
> // checkpoints is enabled
> return (enableCommitOnCheckpoint)
> ? OffsetCommitMode.ON_CHECKPOINTS
> : OffsetCommitMode.DISABLED;
> } else {
> // else, the mode depends only on whether auto committing is
> enabled in the provided
> // Kafka properties
> return (enableAutoCommit) ? OffsetCommitMode.KAFKA_PERIODIC :
> OffsetCommitMode.DISABLED;
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol commented on a diff in pull request #21349: [FLINK-28203] Support Maven 3.3+
zentol commented on code in PR #21349:
URL: https://github.com/apache/flink/pull/21349#discussion_r1191232405
##
tools/ci/flink-ci-tools/src/test/java/org/apache/flink/tools/ci/optional/ShadeOptionalCheckerTest.java:
##
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tools.ci.optional;
+
+import org.apache.flink.tools.ci.utils.shared.Dependency;
+import org.apache.flink.tools.ci.utils.shared.DependencyTree;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+class ShadeOptionalCheckerTest {
+private static final String MODULE = "module";
+
+@Test
+void testNonBundledDependencyIsIgnored() {
+final Dependency dependency = createMandatoryDependency("a");
+final Set bundled = Collections.emptySet();
+final DependencyTree dependencyTree = new
DependencyTree().addDirectDependency(dependency);
+
+final Set violations =
+ShadeOptionalChecker.checkOptionalFlags(MODULE, bundled,
dependencyTree);
+
+assertThat(violations).isEmpty();
+}
+
+@Test
+void testNonBundledDependencyIsIgnoredEvenIfOthersAreBundled() {
+final Dependency dependencyA = createMandatoryDependency("a");
+final Dependency dependencyB = createMandatoryDependency("B");
+final Set bundled = Collections.singleton(dependencyB);
+final DependencyTree dependencyTree =
+new DependencyTree()
+.addDirectDependency(dependencyA)
+.addDirectDependency(dependencyB);
+
+final Set violations =
+ShadeOptionalChecker.checkOptionalFlags(MODULE, bundled,
dependencyTree);
+
+assertThat(violations).containsExactly(dependencyB);
+}
+
+@Test
+void testDirectBundledOptionalDependencyIsAccepted() {
+final Dependency dependency = createOptionalDependency("a");
+final Set bundled = Collections.singleton(dependency);
+final DependencyTree dependencyTree = new
DependencyTree().addDirectDependency(dependency);
+
+final Set violations =
+ShadeOptionalChecker.checkOptionalFlags(MODULE, bundled,
dependencyTree);
+
+assertThat(violations).isEmpty();
+}
+
+@Test
+void testDirectBundledDependencyMustBeOptional() {
+final Dependency dependency = createMandatoryDependency("a");
+final Set bundled = Collections.singleton(dependency);
+final DependencyTree dependencyTree = new
DependencyTree().addDirectDependency(dependency);
+
+final Set violations =
+ShadeOptionalChecker.checkOptionalFlags(MODULE, bundled,
dependencyTree);
+
+assertThat(violations).containsExactly(dependency);
+}
+
+@Test
+void testTransitiveBundledOptionalDependencyIsAccepted() {
+final Dependency dependencyA = createMandatoryDependency("a");
+final Dependency dependencyB = createOptionalDependency("b");
+final Set bundled = Collections.singleton(dependencyB);
+final DependencyTree dependencyTree =
+new DependencyTree()
+.addDirectDependency(dependencyA)
+.addTransitiveDependencyTo(dependencyB, dependencyA);
+
+final Set violations =
+ShadeOptionalChecker.checkOptionalFlags(MODULE, bundled,
dependencyTree);
+
+assertThat(violations).isEmpty();
+}
+
+@Test
+void testTransitiveBundledDependencyMustBeOptional() {
+final Dependency dependencyA = createMandatoryDependency("a");
+final Dependency dependencyB = createMandatoryDependency("b");
+final Set bundled = Collections.singleton(dependencyB);
+final DependencyTree dependencyTree =
+new DependencyTree()
+.addDirectDependency(dependencyA)
+.addTransitiveDependencyTo(dependencyB, dependencyA);
+
+final Set violations =
+ShadeOptionalChecker.checkOptionalFlags(MODULE, bundled,
dependencyTree);
+
+
[jira] [Commented] (FLINK-31963) java.lang.ArrayIndexOutOfBoundsException when scaling down with unaligned checkpoints
[ https://issues.apache.org/jira/browse/FLINK-31963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721788#comment-17721788 ] Stefan Richter commented on FLINK-31963: I have a local reproducer as well as a fix, will open a PR once I have written the tests. > java.lang.ArrayIndexOutOfBoundsException when scaling down with unaligned > checkpoints > - > > Key: FLINK-31963 > URL: https://issues.apache.org/jira/browse/FLINK-31963 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.17.0 > Environment: Flink: 1.17.0 > FKO: 1.4.0 > StateBackend: RocksDB(Genetic Incremental Checkpoint & Unaligned Checkpoint > enabled) >Reporter: Tan Kim >Assignee: Stefan Richter >Priority: Critical > Labels: stability > Attachments: image-2023-04-29-02-49-05-607.png, jobmanager_error.txt, > taskmanager_error.txt > > > I'm testing Autoscaler through Kubernetes Operator and I'm facing the > following issue. > As you know, when a job is scaled down through the autoscaler, the job > manager and task manager go down and then back up again. > When this happens, an index out of bounds exception is thrown and the state > is not restored from a checkpoint. > [~gyfora] told me via the Flink Slack troubleshooting channel that this is > likely an issue with Unaligned Checkpoint and not an issue with the > autoscaler, but I'm opening a ticket with Gyula for more clarification. > Please see the attached JM and TM error logs. > Thank you. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] MartijnVisser commented on a diff in pull request #22546: [FLINK-32032] Upgrade to flink-shaded 17.0
MartijnVisser commented on code in PR #22546: URL: https://github.com/apache/flink/pull/22546#discussion_r1191208612 ## pom.xml: ## @@ -122,8 +122,8 @@ under the License. 4 true -XX:+UseG1GC -Xms256m - 16.1 - 2.13.4 Review Comment: Ah yes, that's what I meant but it doesn't help if I type `4.0` instead of `5.4.0` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32047) Fix args in JobSpec not being passed through to Flink in Standalone mode - 1.4.0
[
https://issues.apache.org/jira/browse/FLINK-32047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721784#comment-17721784
]
Gyula Fora commented on FLINK-32047:
cc[~darenwkt] [~usamj] have you encountered this?
> Fix args in JobSpec not being passed through to Flink in Standalone mode -
> 1.4.0
>
>
> Key: FLINK-32047
> URL: https://issues.apache.org/jira/browse/FLINK-32047
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gil Shmaya
>Priority: Major
> Attachments: image-2023-04-30-18-54-22-291.png,
> image-2023-04-30-19-56-30-150.png, image-2023-04-30-19-56-57-680.png
>
>
> This issue is related to a previously fixed bug in version 1.2.0 -
> FLINK-29388
> I have noticed that while the args are successfully being passed when using
> version 1.2.0, this is not the case with version 1.4.0.
> {+}Scenario{+}:
> I added a log that prints the argument array length at the beginning of the
> main function of the flink job:
> !image-2023-04-30-18-54-22-291.png!
> The result when running with 1.2.0:
> !image-2023-04-30-19-56-30-150.png!
> The result when running with 1.4.0:
> !image-2023-04-30-19-56-57-680.png!
> h4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32047) Fix args in JobSpec not being passed through to Flink in Standalone mode - 1.4.0
[
https://issues.apache.org/jira/browse/FLINK-32047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721782#comment-17721782
]
Gyula Fora commented on FLINK-32047:
If you have time and can work on this, I would be happy to to assign this to
you.[~gil_shmaya]
> Fix args in JobSpec not being passed through to Flink in Standalone mode -
> 1.4.0
>
>
> Key: FLINK-32047
> URL: https://issues.apache.org/jira/browse/FLINK-32047
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gil Shmaya
>Priority: Major
> Attachments: image-2023-04-30-18-54-22-291.png,
> image-2023-04-30-19-56-30-150.png, image-2023-04-30-19-56-57-680.png
>
>
> This issue is related to a previously fixed bug in version 1.2.0 -
> FLINK-29388
> I have noticed that while the args are successfully being passed when using
> version 1.2.0, this is not the case with version 1.4.0.
> {+}Scenario{+}:
> I added a log that prints the argument array length at the beginning of the
> main function of the flink job:
> !image-2023-04-30-18-54-22-291.png!
> The result when running with 1.2.0:
> !image-2023-04-30-19-56-30-150.png!
> The result when running with 1.4.0:
> !image-2023-04-30-19-56-57-680.png!
> h4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-playgrounds] alpinegizmo commented on pull request #42: Create flink_data volume for operations playground.
alpinegizmo commented on PR #42: URL: https://github.com/apache/flink-playgrounds/pull/42#issuecomment-1544024557 Apologies; I think you're right. I was thinking that the playground docs instruct the user to manipulate the savepoint directly on their computer -- but they don't, and for good reason. Instead, this is all done in the client container. I'm referring to this section of the docs -- https://nightlies.apache.org/flink/flink-docs-master/docs/try-flink/flink-operations-playground/#upgrading--rescaling-a-job -- where the user takes a savepoint and then uses it for restarting and rescaling. However, for that to work, I don't think this PR goes far enough. The volume needs to be mounted in the client as well. Could you test that part of the playground docs with this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
[
https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721776#comment-17721776
]
Lijie Wang commented on FLINK-31706:
I think it's a good idea to use {{paralleism.default}} instread of the
{{execution.batch.adaptive.auto-parallelism.default-source-parallelism}}.
Regarding the parallelism of Source in the adaptive batch scheduler, we also
have some other ideas/actions in plan: dynamically infer the Source paralleism
at runtime (according to the amount of data that Source actually needs to read
after Dynamic Partition Pruning). One possible way is that the source
coordinator can infer the parallelism based on the splits information actually
consumed.
At that time, if the parallelism of Source are not specified by the user, the
source coorinator will be responseible for inferring the parallelism
automatically(if it supports). If the Source does not support inferring
parallelism automatically, {{parallelism.default}} will be used as the
parallelism of the Source. (An initial thought :))
> The default source parallelism should be the same as execution's default
> parallelism under adaptive batch scheduler
> ---
>
> Key: FLINK-31706
> URL: https://issues.apache.org/jira/browse/FLINK-31706
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Currently, the sources need to set
> {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in
> the adaptive batch scheduler mode, otherwise, the source parallelism is only
> 1 by default. A better solution might be set as the default execution
> parallelism if no user configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31963) java.lang.ArrayIndexOutOfBoundsException when scaling down with unaligned checkpoints
[ https://issues.apache.org/jira/browse/FLINK-31963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721777#comment-17721777 ] Piotr Nowojski commented on FLINK-31963: We have managed to reproduce and find the bug. Thank you for reporting the issue and help with analysing [~tanee.kim] and [~masteryhx]. We are now working on fixing it. > java.lang.ArrayIndexOutOfBoundsException when scaling down with unaligned > checkpoints > - > > Key: FLINK-31963 > URL: https://issues.apache.org/jira/browse/FLINK-31963 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.17.0 > Environment: Flink: 1.17.0 > FKO: 1.4.0 > StateBackend: RocksDB(Genetic Incremental Checkpoint & Unaligned Checkpoint > enabled) >Reporter: Tan Kim >Assignee: Stefan Richter >Priority: Critical > Labels: stability > Attachments: image-2023-04-29-02-49-05-607.png, jobmanager_error.txt, > taskmanager_error.txt > > > I'm testing Autoscaler through Kubernetes Operator and I'm facing the > following issue. > As you know, when a job is scaled down through the autoscaler, the job > manager and task manager go down and then back up again. > When this happens, an index out of bounds exception is thrown and the state > is not restored from a checkpoint. > [~gyfora] told me via the Flink Slack troubleshooting channel that this is > likely an issue with Unaligned Checkpoint and not an issue with the > autoscaler, but I'm opening a ticket with Gyula for more clarification. > Please see the attached JM and TM error logs. > Thank you. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31963) java.lang.ArrayIndexOutOfBoundsException when scaling down with unaligned checkpoints
[ https://issues.apache.org/jira/browse/FLINK-31963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reassigned FLINK-31963: -- Assignee: Stefan Richter > java.lang.ArrayIndexOutOfBoundsException when scaling down with unaligned > checkpoints > - > > Key: FLINK-31963 > URL: https://issues.apache.org/jira/browse/FLINK-31963 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.17.0 > Environment: Flink: 1.17.0 > FKO: 1.4.0 > StateBackend: RocksDB(Genetic Incremental Checkpoint & Unaligned Checkpoint > enabled) >Reporter: Tan Kim >Assignee: Stefan Richter >Priority: Critical > Labels: stability > Attachments: image-2023-04-29-02-49-05-607.png, jobmanager_error.txt, > taskmanager_error.txt > > > I'm testing Autoscaler through Kubernetes Operator and I'm facing the > following issue. > As you know, when a job is scaled down through the autoscaler, the job > manager and task manager go down and then back up again. > When this happens, an index out of bounds exception is thrown and the state > is not restored from a checkpoint. > [~gyfora] told me via the Flink Slack troubleshooting channel that this is > likely an issue with Unaligned Checkpoint and not an issue with the > autoscaler, but I'm opening a ticket with Gyula for more clarification. > Please see the attached JM and TM error logs. > Thank you. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] echauchot commented on pull request #22527: [FLINK-31749][hotfix][doc] Update Hadoop formats datastream doc
echauchot commented on PR #22527: URL: https://github.com/apache/flink/pull/22527#issuecomment-1544020098 @MartijnVisser I recently learned what the criteria for hotfix changes are. This PR definitely complies. As we already discussed the removal of hadoop outputFormat for DataStream in the ticket, I can self-merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #22527: [FLINK-31749][hotfix][doc] Update Hadoop formats datastream doc
echauchot commented on PR #22527: URL: https://github.com/apache/flink/pull/22527#issuecomment-1544019874 @MartijnVisser I recently learned what the criteria for hotfix changes are. This PR definitely complies. As we already discussed the removal of hadoop outputFormat for DataStream in the ticket, I can self-merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32045) optimize task deployment performance for large-scale jobs
[ https://issues.apache.org/jira/browse/FLINK-32045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721774#comment-17721774 ] Weihua Hu commented on FLINK-32045: --- Thanks [~Thesharing] for your reply, your comment are very meaningful and valuable. Let me try to answer one by one. h3. Distribution of shuffle descriptors via blob server. IMO, there are two things should considered to whether enable distribution of shuffle descriptors via blob server. # The size of shuffle descriptors. This is related to the parallelism of producers for a single consumer. # How many times should this shuffle descriptors transport to TaskManager. This is related to the parallelism of consumers for this producer. So, I think it is better to use the number of edges in ConsumedPartitionGroup to decide whether to enable blob server offload. And I'd like to make this logic internally (give a proper default value, for example 1000*1000, should be decided after some benchmark) since it really needs advanced knowledge for users to figure it out how to set it. h3. how much performance it would improve with a cache for shuffle descriptors in the TaskManager. I have tests in this environment. Yarn cluster with 2000 TaskManager. Each TaskManager has 6 core and 16GB memory and set "taskmanager.numberOfTaskSlots" to 10. Submit a simple WordCount with 2 parallelism. * Without blob server offload, the job failed with submitTask RPC timeout. All CPU of JobManager used to serialized RPC:submitTask. * With blob server offload but no TaskExecutor cache, deploy all tasks take 25s * With blob server offload and TaskExecutor cache, deploy all tasks take 15s h3. How to update the cache? As you mentioned, it's too complicated to keep cache in JobManager and TaskExecutor consistent. So, we will add some constraints to the Cache # Cache will be enabled when necessary (same conditions with distribution of shuffle descriptors via blob server). In most cases serialized shuffle descriptors are small and transport in akka message, the cost of deserialization is very small, they do not need to be cached. # Cache of job will be cleared when task executor disconnects with job master. # Cache with TTL. We should configure a proper default ttl value, for example 3 mins (some batch job may deploy lazily) # The max size of cache. As you mentioned, LRUCache or FIFOCache is reasonable. Since the slots of a Task Manager won't be too large, the cache size won't be too large either. For session mode(more exactly OLAP). IMO, Most of the scenarios are a lot of small queries. As mentioned above, they won't use cache in most cases. And the cache will be removed when the job is finished( task executor disconnects with job master), so the cache won't occupy too much memory in a short time. Thanks again. Also thanks to [~zhuzh] ,[~wanglijie] , [~Weijie Guo] for the previous offline discussions. Glad to hear any suggestions. > optimize task deployment performance for large-scale jobs > - > > Key: FLINK-32045 > URL: https://issues.apache.org/jira/browse/FLINK-32045 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Weihua Hu >Priority: Major > > h1. Background > In FLINK-21110, we cache shuffle descriptors on the job manager side and > support using blob servers to offload these descriptors in order to reduce > the cost of tasks deployment. > I think there is also some improvement we could do for large-scale jobs. > # The default min size to enable distribution via blob server is 1MB. But > for a large wordcount job with 2 parallelism, the size of serialized > shuffle descriptors is only 300KB. It means users need to lower the > "blob.offload.minsize", but the value is hard for users to decide. > # The task executor side still needs to load blob files and deserialize > shuffle descriptors for each task. Since these operations are running in the > main thread, it may be pending other RPCs from the job manager. > h1. Propose > # Enable distribute shuffle descriptors via blob server automatically. This > could be decided by the edge number of the current shuffle descriptor. The > blob offload will be enabled when the edge number exceeds an internal > threshold. > # Introduce cache of deserialized shuffle descriptors on the task executor > side. This could reduce the cost of reading from local blob files and > deserialization. Of course, the cache should have TTL to avoid occupying too > much memory. And the cache should have the same switch mechanism as the blob > server offload. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] echauchot merged pull request #22569: [FLINK-32014][doc][hotfix] Add Cassandra source documentation to website
echauchot merged PR #22569: URL: https://github.com/apache/flink/pull/22569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32047) Fix args in JobSpec not being passed through to Flink in Standalone mode - 1.4.0
[
https://issues.apache.org/jira/browse/FLINK-32047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721770#comment-17721770
]
Gil Shmaya commented on FLINK-32047:
[~gyfora]
Yes, the results are the same:
*!image-2023-05-11-16-21-24-547.png!*
Have you done a change that should resolve that?
This bug blocks us for a long time.
> Fix args in JobSpec not being passed through to Flink in Standalone mode -
> 1.4.0
>
>
> Key: FLINK-32047
> URL: https://issues.apache.org/jira/browse/FLINK-32047
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Gil Shmaya
>Priority: Major
> Attachments: image-2023-04-30-18-54-22-291.png,
> image-2023-04-30-19-56-30-150.png, image-2023-04-30-19-56-57-680.png
>
>
> This issue is related to a previously fixed bug in version 1.2.0 -
> FLINK-29388
> I have noticed that while the args are successfully being passed when using
> version 1.2.0, this is not the case with version 1.4.0.
> {+}Scenario{+}:
> I added a log that prints the argument array length at the beginning of the
> main function of the flink job:
> !image-2023-04-30-18-54-22-291.png!
> The result when running with 1.2.0:
> !image-2023-04-30-19-56-30-150.png!
> The result when running with 1.4.0:
> !image-2023-04-30-19-56-57-680.png!
> h4.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] echauchot commented on pull request #22569: [FLINK-32014][doc][hotfix] Add Cassandra source documentation to website
echauchot commented on PR #22569: URL: https://github.com/apache/flink/pull/22569#issuecomment-1544003128 > ## CI report: > * [68479de](https://github.com/apache/flink/commit/68479de641f3a0f8979d1c270aad6d829025d720) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=48919) > > Bot commands Unrelated failure, merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31743) Avoid relocating the RocksDB's log failure when filename exceeds 255 characters
[ https://issues.apache.org/jira/browse/FLINK-31743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721759#comment-17721759 ] Yun Tang commented on FLINK-31743: -- merged in flink release-1.17: c29af139a8f3055c96c641016a31cd3a92ca022a release-1.16: 7b41bd2b6df4f4794b0b66b52a6af2693a299ca9 > Avoid relocating the RocksDB's log failure when filename exceeds 255 > characters > --- > > Key: FLINK-31743 > URL: https://issues.apache.org/jira/browse/FLINK-31743 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.16.1, 1.15.4 >Reporter: jinghaihang >Assignee: Feifan Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.16.2, 1.18.0, 1.17.1 > > > Since FLINK-24785 , the file name of the rocksdb LOG is generated by parsing > the db path, when the db path is long and the filename exceeds 255 > characters, the creation of the file will fail, so the relevant rocksdb LOG > cannot be seen in the flink log dir. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Myasuka merged pull request #22545: [FLINK-31743][statebackend/rocksdb] disable rocksdb log relocating wh…
Myasuka merged PR #22545: URL: https://github.com/apache/flink/pull/22545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31449) Remove DeclarativeSlotManager related logic
[ https://issues.apache.org/jira/browse/FLINK-31449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu updated FLINK-31449: -- Parent: (was: FLINK-31439) Issue Type: Improvement (was: Sub-task) > Remove DeclarativeSlotManager related logic > --- > > Key: FLINK-31449 > URL: https://issues.apache.org/jira/browse/FLINK-31449 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Weihua Hu >Priority: Blocker > > The DeclarativeSlotManager and related configs will be completely removed in > the next release after the default SlotManager change to > FineGrainedSlotManager. > > We should do the job in 1.19 version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31537) Derive a new TaskManagerTrackerConfiguration from the SlotManagerConfiguration.
[ https://issues.apache.org/jira/browse/FLINK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu updated FLINK-31537: -- Parent: (was: FLINK-31439) Issue Type: Improvement (was: Sub-task) > Derive a new TaskManagerTrackerConfiguration from the > SlotManagerConfiguration. > --- > > Key: FLINK-31537 > URL: https://issues.apache.org/jira/browse/FLINK-31537 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Weihua Hu >Priority: Major > > As discussion in > https://github.com/apache/flink/pull/22196#discussion_r1141847905 > We need derive a new TaskManagerTrackerConfiguration from the > SlotManagerConfiguration after DeclarativeSlotManager removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31445) Split resource allocate/release related logic from FineGrainedSlotManager to TaskManagerTracker
[ https://issues.apache.org/jira/browse/FLINK-31445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu updated FLINK-31445: -- Parent: (was: FLINK-31439) Issue Type: Improvement (was: Sub-task) > Split resource allocate/release related logic from FineGrainedSlotManager to > TaskManagerTracker > --- > > Key: FLINK-31445 > URL: https://issues.apache.org/jira/browse/FLINK-31445 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Weihua Hu >Assignee: Weihua Hu >Priority: Major > Labels: pull-request-available > > Currently the FineGrainedSlotManager is response to slots allocations and > resources request/release. This makes the logical of FineGrainedSlotManager > complicated, So we will move task manager related work from > FineGrainedSlotManager to TaskManagerTracker, which already tracks task > managers but not including request/release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27925) Avoid to create watcher without the resourceVersion
[ https://issues.apache.org/jira/browse/FLINK-27925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721757#comment-17721757 ] Weihua Hu commented on FLINK-27925: --- Thanks [~ouyangwulin] , glad to hear that. > Avoid to create watcher without the resourceVersion > --- > > Key: FLINK-27925 > URL: https://issues.apache.org/jira/browse/FLINK-27925 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: Aitozi >Assignee: ouyangwulin >Priority: Major > Labels: pull-request-available > Attachments: image-2022-12-19-20-19-41-303.png > > > Currently, we create the watcher in KubernetesResourceManager. But it do not > pass the resourceVersion parameter, it will trigger a request to etcd. It > will bring the burden to the etcd in large scale cluster (which have been > seen in our internal k8s cluster). More detail can be found > [here|https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter] > > I think we could use the informer to improve it (which will spawn a > list-watch and maintain the resourceVersion internally) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31439) FLIP-298: Unifying the Implementation of SlotManager
[ https://issues.apache.org/jira/browse/FLINK-31439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-31439: --- Affects Version/s: 1.18.0 > FLIP-298: Unifying the Implementation of SlotManager > > > Key: FLINK-31439 > URL: https://issues.apache.org/jira/browse/FLINK-31439 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.18.0 >Reporter: Weihua Hu >Assignee: Weihua Hu >Priority: Major > Fix For: 1.18.0 > > > This is an umbrella ticket for > [FLIP-298|https://cwiki.apache.org/confluence/display/FLINK/FLIP-298%3A+Unifying+the+Implementation+of+SlotManager]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-web] gyfora opened a new pull request, #647: Kubernetes Operator 1.5.0
gyfora opened a new pull request, #647: URL: https://github.com/apache/flink-web/pull/647 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31967) SQL with LAG function NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721756#comment-17721756
]
Shengkai Fang commented on FLINK-31967:
---
[~padavan] after investigation, the type inference is not correct for the lag
function. For a quick fix, you can modify the type of the `count` in the
UserModel to `Integer`.
> SQL with LAG function NullPointerException
> --
>
> Key: FLINK-31967
> URL: https://issues.apache.org/jira/browse/FLINK-31967
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: padavan
>Priority: Major
> Attachments: image-2023-04-28-14-46-19-736.png,
> image-2023-04-28-15-06-48-184.png, image-2023-04-28-15-14-58-788.png,
> image-2023-04-28-15-17-49-144.png, image-2023-04-28-17-06-20-737.png,
> simpleFlinkKafkaLag.zip
>
>
> I want to make a query with the LAG function. And got Job Exception without
> any explanations.
>
> *Code:*
> {code:java}
> private static void t1_LeadLag(DataStream ds,
> StreamExecutionEnvironment env) {
> StreamTableEnvironment te = StreamTableEnvironment.create(env);
> Table t = te.fromDataStream(ds,
> Schema.newBuilder().columnByExpression("proctime", "proctime()").build());
> te.createTemporaryView("users", t);
> Table res = te.sqlQuery("SELECT userId, `count`,\n" +
> " LAG(`count`) OVER (PARTITION BY userId ORDER BY proctime) AS
> prev_quantity\n" +
> " FROM users");
> te.toChangelogStream(res).print();
> }{code}
>
> *Input:*
> {"userId":3,"count":0,"dt":"2023-04-28T07:44:21.551Z"}
>
> *Exception:* I remove part about basic JobExecutionException and kept the
> important(i think)
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.table.data.GenericRowData.getInt(GenericRowData.java:149)
> at
> org.apache.flink.table.data.RowData.lambda$createFieldGetter$245ca7d1$6(RowData.java:245)
> at
> org$apache$flink$table$runtime$functions$aggregate$LagAggFunction$LagAcc$2$Converter.toExternal(Unknown
> Source)
> at
> org.apache.flink.table.data.conversion.StructuredObjectConverter.toExternal(StructuredObjectConverter.java:101)
> at UnboundedOverAggregateHelper$15.setAccumulators(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.over.ProcTimeUnboundedPrecedingFunction.processElement(ProcTimeUnboundedPrecedingFunction.java:92)
> at
> org.apache.flink.table.runtime.operators.over.ProcTimeUnboundedPrecedingFunction.processElement(ProcTimeUnboundedPrecedingFunction.java:42)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Thread.java:829){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] Myasuka commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
Myasuka commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1191127181
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java:
##
@@ -394,18 +394,14 @@ public void confirm(SequenceNumber from, SequenceNumber
to, long checkpointId) {
.forEach(
localHandle -> {
changelogRegistry.stopTracking(localHandle);
-localChangelogRegistry.register(localHandle,
checkpointId);
});
-}
-
-@Override
-public void subsume(long checkpointId) {
localChangelogRegistry.discardUpToCheckpoint(checkpointId);
}
@Override
public void reset(SequenceNumber from, SequenceNumber to, long
checkpointId) {
-localChangelogRegistry.prune(checkpointId);
+// delete all accumulated local dstl files when abort
+localChangelogRegistry.discardUpToCheckpoint(checkpointId + 1);
Review Comment:
Thanks for the clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Myasuka commented on pull request #22555: [FLINK-31706] [runtime] The default source parallelism should be the same as ex…
Myasuka commented on PR #22555: URL: https://github.com/apache/flink/pull/22555#issuecomment-1543921566 @clownxc Please move to the original ticket. Any changes to break the previous behavior needs a discussion in the JIRA, that's why we did not assign anyone to take the ticket. BTW, we think this change should be reasonable indeed, but a discussion is necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-31903) Caching records fails in BroadcastUtils#withBroadcastStream
[
https://issues.apache.org/jira/browse/FLINK-31903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhipeng Zhang closed FLINK-31903.
-
Resolution: Not A Bug
> Caching records fails in BroadcastUtils#withBroadcastStream
> ---
>
> Key: FLINK-31903
> URL: https://issues.apache.org/jira/browse/FLINK-31903
> Project: Flink
> Issue Type: Bug
> Components: Library / Machine Learning
>Affects Versions: ml-2.3.0
>Reporter: Zhipeng Zhang
>Priority: Major
>
> When caching more than 1,000,000 records using BroadcastUtils#withBroadcast,
> it throws exception as follows:
> {code:java}
> Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint
> tolerable failure threshold.
> at
> org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:206)
> at
> org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:191)
> at
> org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:124)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2078)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1038)
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$declineCheckpoint$2(ExecutionGraphHandler.java:103)
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> It seems that the bug comes from caching too many records when calling
> AbstractBroadcastWrapperOperator#snapshot.
>
> The failed case could be found here:
> [https://github.com/zhipeng93/flink-ml/tree/FLINK-31903-fail-case]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31903) Caching records fails in BroadcastUtils#withBroadcastStream
[
https://issues.apache.org/jira/browse/FLINK-31903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721753#comment-17721753
]
Zhipeng Zhang commented on FLINK-31903:
---
The root cause is that using memory as statebackend has limited state size. So
this is not a bug.
" java.io.IOException: Size of the state is larger than the maximum permitted
memory-backed state. Size=5365814, maxSize=5242880. Consider using a different
checkpoint storage, like the FileSystemCheckpointStorage.".
> Caching records fails in BroadcastUtils#withBroadcastStream
> ---
>
> Key: FLINK-31903
> URL: https://issues.apache.org/jira/browse/FLINK-31903
> Project: Flink
> Issue Type: Bug
> Components: Library / Machine Learning
>Affects Versions: ml-2.3.0
>Reporter: Zhipeng Zhang
>Priority: Major
>
> When caching more than 1,000,000 records using BroadcastUtils#withBroadcast,
> it throws exception as follows:
> {code:java}
> Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint
> tolerable failure threshold.
> at
> org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:206)
> at
> org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:191)
> at
> org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:124)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2078)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1038)
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$declineCheckpoint$2(ExecutionGraphHandler.java:103)
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> It seems that the bug comes from caching too many records when calling
> AbstractBroadcastWrapperOperator#snapshot.
>
> The failed case could be found here:
> [https://github.com/zhipeng93/flink-ml/tree/FLINK-31903-fail-case]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] Mrart commented on a diff in pull request #21527: [FLINK-27925] [kubernetes]Performance optimization when watch tm pod and list pod.
Mrart commented on code in PR #21527:
URL: https://github.com/apache/flink/pull/21527#discussion_r1191070958
##
flink-kubernetes/src/test/java/org/apache/flink/kubernetes/KubernetesClientTestBase.java:
##
@@ -143,6 +147,34 @@ protected void mockGetDeploymentWithError() {
server.expect().get().withPath(path).andReturn(500, "Expected
error").always();
}
+protected void mockPodEventWithLabels(Map labels) {
+final Pod pod1 =
+new PodBuilder()
+.withNewMetadata()
+.withNamespace("test")
+.withName("tm_pod1")
+.withLabels(labels)
+.withResourceVersion("5668")
+.endMetadata()
+.build();
+// mock four kinds of events.
+server.expect()
+.withPath(
+
"/api/v1/namespaces/test/pods?labelSelector=label1%3Dvalue1%2Clabel2%3Dvalue2=0=true=true")
+.andUpgradeToWebSocket()
+.open()
+.waitFor(1000)
Review Comment:
definitely wait for 1 second
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Mrart commented on a diff in pull request #21527: [FLINK-27925] [kubernetes]Performance optimization when watch tm pod and list pod.
Mrart commented on code in PR #21527:
URL: https://github.com/apache/flink/pull/21527#discussion_r1191067760
##
flink-kubernetes/src/test/java/org/apache/flink/kubernetes/KubernetesClientTestBase.java:
##
@@ -143,6 +147,34 @@ protected void mockGetDeploymentWithError() {
server.expect().get().withPath(path).andReturn(500, "Expected
error").always();
}
+protected void mockPodEventWithLabels(Map labels) {
+final Pod pod1 =
+new PodBuilder()
+.withNewMetadata()
+.withNamespace("test")
+.withName("tm_pod1")
+.withLabels(labels)
+.withResourceVersion("5668")
+.endMetadata()
+.build();
+// mock four kinds of events.
+server.expect()
+.withPath(
+
"/api/v1/namespaces/test/pods?labelSelector=label1%3Dvalue1%2Clabel2%3Dvalue2=0=true=true")
+.andUpgradeToWebSocket()
+.open()
+.waitFor(1000)
Review Comment:
The waitFor parameter specifies the amount of time the program should wait
for the expected output to appear before timing out. If the expected output
does not appear within the specified time, the program will raise a timeout
error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-32063) AWS CI mvn compile fails to cast objects to parent type.
Ahmed Hamdy created FLINK-32063:
---
Summary: AWS CI mvn compile fails to cast objects to parent type.
Key: FLINK-32063
URL: https://issues.apache.org/jira/browse/FLINK-32063
Project: Flink
Issue Type: Bug
Components: Connectors / AWS, Tests
Reporter: Ahmed Hamdy
h2. Description
AWS Connectors CI fails to cast {{TestSinkInitContext}} into base type
{{InitContext}},
- Failure
https://github.com/apache/flink-connector-aws/actions/runs/4924790308/jobs/8841458606?pr=70
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
