harsh61426 opened a new issue #722: URL: https://github.com/apache/openwhisk-deploy-kube/issues/722
Im unable to get the Controller and Invoker to running 1/1 state.
$kubectl get pods -n openwhisk
NAME READY STATUS RESTARTS AGE
owdev-alarmprovider-6c89d8846b-rjpqf 1/1 Running 0 95d
owdev-apigateway-d6d756db8-x7kb4 1/1 Running 0 95d
owdev-controller-0 0/1 Running 15 3h29m
owdev-couchdb-5666bfc5f6-gnktm 1/1 Running 0 95d
owdev-gen-certs-649zd 0/1 Completed 0 95d
owdev-init-couchdb-f5g6k 0/1 Completed 0 95d
owdev-install-packages-7psb4 0/1 Completed 0 95d
owdev-invoker-0 0/1 Init:0/1 0 4h1m
owdev-kafka-0 1/1 Running 0 4h10m
owdev-kafkaprovider-7867778f74-rhhzt 1/1 Running 0 95d
owdev-nginx-67b8974c77-dcwt4 1/1 Running 0 95d
owdev-redis-cdb68fc8f-tdczh 1/1 Running 0 95d
owdev-wskadmin 1/1 Running 0 95d
owdev-zookeeper-0 1/1 Running 0 4h10m
$ kubectl logs owdev-controller-0 -n openwhisk
[2022-02-14T06:53:43.612Z] [INFO] Slf4jLogger started
[2022-02-14T06:53:44.040Z] [WARN] Failed to attach the instrumentation
because the Kamon Bundle is not present on the classpath
[2022-02-14T06:53:44.196Z] [INFO] Started the Kamon StatsD reporter
[2022-02-14T06:53:44.504Z] [INFO] [#tid_sid_unknown] [Config] environment
set value for limits.triggers.fires.perMinute
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment
set value for limits.actions.sequence.maxLength
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment
set value for limits.actions.invokes.concurrent
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment
set value for limits.actions.invokes.perMinute
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment
set value for runtimes.manifest
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment
set value for kafka.hosts
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment
set value for port
[2022-02-14T06:53:45.186Z] [INFO] [#tid_sid_unknown]
[KafkaMessagingProvider] completed0 already exists and the user can see it,
skipping creation.
[2022-02-14T06:53:45.366Z] [INFO] [#tid_sid_unknown]
[KafkaMessagingProvider] health already exists and the user can see it,
skipping creation.
[2022-02-14T06:53:45.534Z] [INFO] [#tid_sid_unknown]
[KafkaMessagingProvider] cacheInvalidation already exists and the user can see
it, skipping creation.
[2022-02-14T06:53:45.678Z] [INFO] [#tid_sid_unknown]
[KafkaMessagingProvider] events already exists and the user can see it,
skipping creation.
[2022-02-14T06:53:45.871Z] [INFO] [#tid_sid_controller] [Controller]
starting controller instance 0 [marker:controller_startup0_counter:1393]
[2022-02-14T06:53:46.527Z] [INFO] [#tid_sid_dispatcher] [MessageFeed]
handler capacity = 128, pipeline fill at = 128, pipeline depth = 256
[2022-02-14T06:53:46.718Z] [INFO] [#tid_sid_loadbalancer]
[ShardingContainerPoolBalancerState] managedFraction = 0.9, blackboxFraction =
0.1
[2022-02-14T06:53:46.782Z] [INFO] [#tid_sid_dispatcher] [MessageFeed]
handler capacity = 128, pipeline fill at = 128, pipeline depth = 256
[2022-02-14T06:53:47.131Z] [INFO] [#tid_sid_loadbalancer] [WhiskAction]
[GET] serving from datastore: CacheKey(whisk.system/invokerHealthTestAction0)
[marker:database_cacheMiss_counter:2654]
[2022-02-14T06:53:47.183Z] [INFO] [#tid_sid_loadbalancer] [CouchDbRestStore]
[GET] 'test_whisks' finding document: 'id:
whisk.system/invokerHealthTestAction0' [marker:database_getDocument_start:2706]
[2022-02-14T06:53:47.376Z] [INFO] [#tid_sid_loadbalancer] [CouchDbRestStore]
[marker:database_getDocument_finish:2899:192]
[2022-02-14T06:53:47.427Z] [INFO] [#tid_sid_loadbalancer] [WhiskAction]
invalidating CacheKey(whisk.system/invokerHealthTestAction0)
Exception in thread "main"
org.apache.openwhisk.core.database.DocumentTypeMismatchException: document type
class org.apache.openwhisk.core.entity.WhiskTrigger did not match expected type
class org.apache.openwhisk.core.entity.WhiskAction.
[2022-02-14T06:53:47.432Z] [ERROR] [#tid_sid_loadbalancer] [InvokerPool]
error creating test action for invoker health:
org.apache.openwhisk.core.database.DocumentTypeMismatchException: document type
class org.apache.openwhisk.core.entity.WhiskTrigger did not match expected type
class org.apache.openwhisk.core.entity.WhiskAction.
at
org.apache.openwhisk.core.database.StoreUtils$.deserialize(StoreUtils.scala:71)
at
org.apache.openwhisk.core.database.CouchDbRestStore.$anonfun$get$5(CouchDbRestStore.scala:233)
at
org.apache.openwhisk.core.database.CouchDbRestStore$$Lambda$1297/0000000058052500.apply(Unknown
Source)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.Future$$Lambda$658/00000000A0007890.apply(Unknown
Source)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at
scala.concurrent.impl.Promise$$Lambda$164/00000000091939C0.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
at
akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$1131/000000007C0031D0.apply$mcV$sp(Unknown
Source)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:836)
$ kubectl describe pod owdev-controller-0 -n openwhisk
Name: owdev-controller-0
Namespace: openwhisk
Priority: 0
Node: dvorak-2-3/10.1.0.23
Start Time: Sun, 13 Feb 2022 21:36:04 -0600
Labels: app=owdev-openwhisk
chart=openwhisk-1.0.1
controller-revision-hash=owdev-controller-c6c445889
heritage=Helm
name=owdev-controller
release=owdev
statefulset.kubernetes.io/pod-name=owdev-controller-0
Annotations: <none>
Status: Running
IP: 10.244.1.227
IPs:
IP: 10.244.1.227
Controlled By: StatefulSet/owdev-controller
Init Containers:
wait-for-kafka:
Container ID:
docker://6227eb6e184643a13a14256caf5f9a3c1c05f11c5300a4cc329013cc86772326
Image: openwhisk/ow-utils:3e6138d
Image ID:
docker-pullable://openwhisk/ow-utils@sha256:0b1a3531ade6774069f3f5777784f1d2446d2165537dd617bbc7302bcfb690b1
Port: <none>
Host Port: <none>
Command:
sh
-c
cacert="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt";
token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"; while true;
do rc=$(curl -sS --cacert $cacert --header "Authorization: Bearer $token"
https://kubernetes.default.svc/api/v1/namespaces/openwhisk/endpoints/owdev-kafka
| jq -r ".subsets[].addresses | length"); echo "num ready kafka endpoints is
$rc"; if [ $rc -gt 0 ]; then echo "Success: ready kafka endpoint!"; break; fi;
echo "kafka not ready yet; sleeping for 3 seconds"; sleep 3; done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 13 Feb 2022 21:36:06 -0600
Finished: Sun, 13 Feb 2022 21:36:06 -0600
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from
owdev-core-token-4ptj9 (ro)
wait-for-couchdb:
Container ID:
docker://eb91335d1e63d2eeb74bb37d3509f026fb2157d76e896a4c1bfb0e46ba1ec0d3
Image: busybox:latest
Image ID:
docker-pullable://busybox@sha256:139abcf41943b8bcd4bc5c42ee71ddc9402c7ad69ad9e177b0a9bc4541f14924
Port: <none>
Host Port: <none>
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider
$READINESS_URL --header="Authorization: Basic
d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then
echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping
3 seconds before retry'; sleep 3; done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 13 Feb 2022 21:36:07 -0600
Finished: Sun, 13 Feb 2022 21:36:07 -0600
Ready: True
Restart Count: 0
Environment:
READINESS_URL:
http://owdev-couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from
owdev-core-token-4ptj9 (ro)
Containers:
controller:
Container ID:
docker://50bf9291af60ceaaa609157a376819b13469cc17ed3b71681033f2651ef8c51d
Image: openwhisk/controller:3e6138d
Image ID:
docker-pullable://openwhisk/controller@sha256:51e0f338a942fe395358263cf5f76f673bb949b3d36494556fac760a22df899a
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh `hostname | awk -F '-' '{print $NF}'`
State: Running
Started: Mon, 14 Feb 2022 00:53:41 -0600
Last State: Terminated
Reason: Error
Exit Code: 143
Started: Mon, 14 Feb 2022 00:40:21 -0600
Finished: Mon, 14 Feb 2022 00:53:40 -0600
Ready: False
Restart Count: 15
Liveness: http-get http://:8080/ping delay=200s timeout=600s
period=200s #success=1 #failure=3
Readiness: http-get http://:8080/ping delay=200s timeout=600s
period=200s #success=1 #failure=3
Environment:
PORT: 8080
TZ: UTC
CONFIG_whisk_info_date: <set to
the key 'whisk_info_date' of config map 'owdev-whisk.config'> Optional:
false
CONFIG_whisk_info_buildNo: <set to
the key 'whisk_info_buildNo' of config map 'owdev-whisk.config'> Optional:
false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs:12",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v12",
"tag": "1.19.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "nodejs:14",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v14",
"tag": "1.19.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"stemCells": [
{
"initialCount": 2,
"memory": "256 MB",
"reactive": {
"minCount": 1,
"maxCount": 4,
"ttl": "2 minutes",
"threshold": 1,
"increment": 1
}
}
]
}
],
"python": [
{
"kind": "python:3",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-python-v3.7",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"swift": [
{
"kind": "swift:4.2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.2",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "swift:5.1",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v5.1",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "swift:5.3",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v5.3",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "swift:5.4",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v5.4",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"java": [
{
"kind": "java:8",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"requireMain": true
}
],
"php": [
{
"kind": "php:7.3",
"default": false,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.3",
"tag": "1.17.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "php:7.4",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.4",
"tag": "1.17.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "php:8.0",
"default": false,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.4",
"tag": "1.17.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "1.17.0"
}
}
],
"go": [
{
"kind": "go:1.15",
"default": true,
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"image": {
"prefix": "openwhisk",
"name": "action-golang-v1.15",
"tag": "1.18.0"
}
}
],
"rust": [
{
"kind": "rust:1.34",
"default": true,
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"image": {
"prefix": "openwhisk",
"name": "action-rust-v1.34",
"tag": "1.3.0"
}
}
],
"dotnet": [
{
"kind": "dotnet:2.2",
"default": true,
"deprecated": false,
"requireMain": true,
"image": {
"prefix": "openwhisk",
"name": "action-dotnet-v2.2",
"tag": "1.16.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "dotnet:3.1",
"default": false,
"deprecated": false,
"requireMain": true,
"image": {
"prefix": "openwhisk",
"name": "action-dotnet-v3.1",
"tag": "1.16.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"ballerina": [
{
"kind": "ballerina:0.990",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-ballerina-v0.990.2",
"tag": "nightly"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "1.14.0"
}
]
}
LIMITS_ACTIONS_INVOKES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_CONCURRENT: 30
LIMITS_TRIGGERS_FIRES_PERMINUTE: 60
LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50
CONFIG_whisk_timeLimit_min: 100ms
CONFIG_whisk_timeLimit_max: 5m
CONFIG_whisk_timeLimit_std: 1m
CONFIG_whisk_memory_min: 128m
CONFIG_whisk_memory_max: 512m
CONFIG_whisk_memory_std: 256m
CONFIG_whisk_concurrencyLimit_min: 1
CONFIG_whisk_concurrencyLimit_max: 1
CONFIG_whisk_concurrencyLimit_std: 1
CONFIG_whisk_logLimit_min: 0m
CONFIG_whisk_logLimit_max: 10m
CONFIG_whisk_logLimit_std: 10m
CONFIG_whisk_activation_payload_max: 1048576
CONFIG_whisk_loadbalancer_blackboxFraction: 10%
CONFIG_whisk_loadbalancer_timeoutFactor: 2
KAFKA_HOSTS:
owdev-kafka-0.owdev-kafka.openwhisk.svc.cluster.local:9092
CONFIG_whisk_kafka_replicationFactor:
CONFIG_whisk_kafka_topics_cacheInvalidation_retentionBytes:
CONFIG_whisk_kafka_topics_cacheInvalidation_retentionMs:
CONFIG_whisk_kafka_topics_cacheInvalidation_segmentBytes:
CONFIG_whisk_kafka_topics_completed_retentionBytes:
CONFIG_whisk_kafka_topics_completed_retentionMs:
CONFIG_whisk_kafka_topics_completed_segmentBytes:
CONFIG_whisk_kafka_topics_events_retentionBytes:
CONFIG_whisk_kafka_topics_events_retentionMs:
CONFIG_whisk_kafka_topics_events_segmentBytes:
CONFIG_whisk_kafka_topics_health_retentionBytes:
CONFIG_whisk_kafka_topics_health_retentionMs:
CONFIG_whisk_kafka_topics_health_segmentBytes:
CONFIG_whisk_kafka_topics_invoker_retentionBytes:
CONFIG_whisk_kafka_topics_invoker_retentionMs:
CONFIG_whisk_kafka_topics_invoker_segmentBytes:
CONFIG_whisk_couchdb_username: <set to
the key 'db_username' in secret 'owdev-db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to
the key 'db_password' in secret 'owdev-db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to
the key 'db_port' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to
the key 'db_protocol' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_host:
owdev-couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to
the key 'db_provider' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to
the key 'db_whisk_activations' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to
the key 'db_whisk_actions' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to
the key 'db_whisk_auths' of config map 'owdev-db.config'> Optional: false
CONTROLLER_INSTANCES: 1
CONFIG_logback_log_level: INFO
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from
owdev-core-token-4ptj9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
owdev-core-token-4ptj9:
Type: Secret (a volume populated by a Secret)
SecretName: owdev-core-token-4ptj9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 8m15s (x45 over 3h24m) kubelet Readiness probe
failed: Get "http://10.244.1.227:8080/ping": dial tcp 10.244.1.227:8080:
connect: connection refused
Warning Unhealthy 2m10s (x47 over 3h25m) kubelet Liveness probe
failed: Get "http://10.244.1.227:8080/ping": dial tcp 10.244.1.227:8080:
connect: connection refused
I increased the initialDelaySeconds for the liveness and readiness probe for
the controller, but it doesn't work.
Can you please tell me how to resolve this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
