[
https://issues.apache.org/jira/browse/FLINK-22006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310715#comment-17310715
]
Yi Tang commented on FLINK-22006:
---------------------------------
Yeah, there are the two configurations is relative, maxRequests and
maxRequestsPerHost. For webSocket, maxRequestsPerHost is ignored[1]. So the
actual effective configuration is maxRequests with default value 64.
And here are two small PoCs running with the corresponding dependency versions,
and you can only get at most 64 ADDED events or 64 open logs.
{code:java}
package org.example;
import io.fabric8.kubernetes.api.model.ConfigMap;
import io.fabric8.kubernetes.client.*;
public class K {
public static void main(String[] args) {
Config config = Config.autoConfigure(null);
// config.setMaxConcurrentRequests(1000);
// config.setMaxConcurrentRequestsPerHost(1000);
final NamespacedKubernetesClient client = new
DefaultKubernetesClient(config);
for (int i = 0; i < 100; i++) {
client.configMaps().inNamespace("default").withName("abc-" +
i).watch(new Watcher<ConfigMap>() {
@Override
public void eventReceived(Action action, ConfigMap configMap) {
System.out.println(action +
configMap.getMetadata().getName());
}
@Override
public void onClose(KubernetesClientException e) {}
});
}
}
}
{code}
{code:java}
package org.example;
import okhttp3.*;
public class T {
public static void main(String[] args) {
OkHttpClient client = new OkHttpClient();
// client.dispatcher().setMaxRequests(1000);
// client.dispatcher().setMaxRequestsPerHost(1000);
for (int i = 0; i < 100; i++) {
String tag = "open" + i;
client.newWebSocket(
new Request.Builder().url("http://127.0.0.1:9999";).build(),
new WebSocketListener() {
@Override
public void onOpen(WebSocket webSocket, Response response) {
super.onOpen(webSocket, response);
System.out.println(tag);
}
});
}
}
}
{code}
And i think it's ok to walk around this issue by setting a larger value for
maxRequests temporarily.
1.
[https://github.com/square/okhttp/blob/parent-3.12.1/okhttp/src/main/java/okhttp3/Dispatcher.java#L196]
> Could not run more than 20 jobs in a native K8s session when K8s HA enabled
> ---------------------------------------------------------------------------
>
> Key: FLINK-22006
> URL: https://issues.apache.org/jira/browse/FLINK-22006
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.12.2, 1.13.0
> Reporter: Yang Wang
> Priority: Critical
> Labels: k8s-ha
> Attachments: image-2021-03-24-18-08-42-116.png
>
>
> Currently, if we start a native K8s session cluster when K8s HA enabled, we
> could not run more than 20 streaming jobs.
>
> The latest job is always initializing, and the previous one is created and
> waiting to be assigned. It seems that some internal resources have been
> exhausted, e.g. okhttp thread pool , tcp connections or something else.
> !image-2021-03-24-18-08-42-116.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
