nicknezis commented on issue #3724:
URL:
https://github.com/apache/incubator-heron/issues/3724#issuecomment-974721926
This looks great. Did we need to make two services? Or did the single
headless service match on the pods from both StatefulSet instances?
Should we also allow for setting both the request and limit values for
manager cpu and memory?
For the newly updated Helm chart, you should be able to configure the UI
service to expose a NodePort (ui.nodeport.enabled). With that, you should
be able to run the "minikube services" command to list and easily view.
(Sorry I don't remember the specifics of that command, but it is a
convenience that minikube provides).
On Sat, Nov 20, 2021 at 2:00 PM Saad Ur Rahman ***@***.***>
wrote:
> The CLI commands to configure the Manager limits:
>
> --config-property heron.kubernetes.manager.limits.cpu=[NATURAL NUMBER
VALUE]
> --config-property heron.kubernetes.manager.limits.memory=[NATURAL NUMBER
VALUE]
>
> Sample commands for submitting:
>
> ~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
> --verbose \
> --config-property heron.kubernetes.manager.limits.cpu=1 \
> --config-property heron.kubernetes.manager.limits.memory=2 \
> org.apache.heron.examples.api.AckingTopology acking
>
> I am not able to connect to the Heron UI when running in Minikube but that
> has always been the case for me. I can confirm that the topology is
> executing using CPU and Memory graph traces.
> Manager StatefulSet
>
> apiVersion: apps/v1kind: StatefulSetmetadata:
> creationTimestamp: "2021-11-20T17:58:18Z"
> generation: 1
> labels:
> app: heron
> topology: acking
> name: acking-manager
> namespace: default
> resourceVersion: "1756"
> uid: 2a5d6ba3-d31f-4e36-997b-984545f7a3a7spec:
> podManagementPolicy: Parallel
> replicas: 1
> revisionHistoryLimit: 10
> selector:
> matchLabels:
> app: heron
> topology: acking
> serviceName: acking
> template:
> metadata:
> annotations:
> prometheus.io/port: "8080"
> prometheus.io/scrape: "true"
> creationTimestamp: null
> labels:
> app: heron
> topology: acking
> spec:
> containers:
> - command:
> - sh
> - -c
> - './heron-core/bin/heron-downloader-config kubernetes &&
./heron-core/bin/heron-downloader
distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--7278444175656031685.tar.gz
. && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} &&
./heron-core/bin/heron-executor --topology-name=acking
--topology-id=acking17797dee-b1f1-4afa-8018-0054ffb2f988
--topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
--state-manager-root=/heron
--state-manager-config-file=./heron-conf/statemgr.yaml
--tmanager-binary=./heron-core/bin/heron-tmanager
--stmgr-binary=./heron-core/bin/heron-stmgr
--metrics-manager-classpath=./heron-core/lib/metricsmgr/*
--instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
--classpath=heron-api-examples.jar
--heron-internals-config-file=./heron-conf/heron_internals.yaml
--override-config-file=./heron-conf/override.yaml --component-ram-map
=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
--pkg-type=jar --topology-binary-file=heron-api-examples.jar
--heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
--cluster=kubernetes --role=saad --environment=default
--instance-classpath=./heron-core/lib/instance/*
--metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
--scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
--python-instance-binary=./heron-core/bin/heron-python-instance
--cpp-instance-binary=./heron-core/bin/heron-cpp-instance
--metricscache-manager-classpath=./heron-core/lib/metricscachemgr/*
--metricscache-manager-mode=disabled --is-stateful=false
--checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
--stateful-config-file=./heron-conf/stateful.yaml
--checkpoint-manager-ram=1073741824 --health-manager-m
ode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
--shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002
--tmanager-stats-port=6003 --shell-port=6004
--metrics-manager-port=6005 --scheduler-port=6006
--metricscache-manager-server-port=6007
--metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
> env:
> - name: HOST
> valueFrom:
> fieldRef:
> apiVersion: v1
> fieldPath: status.podIP
> - name: POD_NAME
> valueFrom:
> fieldRef:
> apiVersion: v1
> fieldPath: metadata.name
> image: apache/heron:testbuild
> imagePullPolicy: IfNotPresent
> name: manager
> ports:
> - containerPort: 6001
> name: server
> protocol: TCP
> - containerPort: 6004
> name: shell-port
> protocol: TCP
> - containerPort: 6002
> name: tmanager-ctl
> protocol: TCP
> - containerPort: 6005
> name: metrics-mgr
> protocol: TCP
> - containerPort: 6009
> name: ckptmgr
> protocol: TCP
> - containerPort: 6007
> name: metrics-cache-m
> protocol: TCP
> - containerPort: 6003
> name: tmanager-stats
> protocol: TCP
> - containerPort: 6008
> name: metrics-cache-s
> protocol: TCP
> - containerPort: 6006
> name: scheduler
> protocol: TCP
> resources:
> limits:
> cpu: "1"
> memory: 2Gi
> terminationMessagePath: /dev/termination-log
> terminationMessagePolicy: File
> dnsPolicy: ClusterFirst
> restartPolicy: Always
> schedulerName: default-scheduler
> securityContext: {}
> terminationGracePeriodSeconds: 0
> tolerations:
> - effect: NoExecute
> key: node.kubernetes.io/not-ready
> operator: Exists
> tolerationSeconds: 10
> - effect: NoExecute
> key: node.kubernetes.io/unreachable
> operator: Exists
> tolerationSeconds: 10
> updateStrategy:
> rollingUpdate:
> partition: 0
> type: RollingUpdatestatus:
> availableReplicas: 1
> collisionCount: 0
> currentReplicas: 1
> currentRevision: acking-manager-7ff7f4fb89
> observedGeneration: 1
> readyReplicas: 1
> replicas: 1
> updateRevision: acking-manager-7ff7f4fb89
> updatedReplicas: 1
>
> Executors StatefulSet
>
> apiVersion: apps/v1kind: StatefulSetmetadata:
> creationTimestamp: "2021-11-20T17:58:18Z"
> generation: 1
> labels:
> app: heron
> topology: acking
> name: acking-executors
> namespace: default
> resourceVersion: "1752"
> uid: d8c9d174-5feb-49a5-9297-c84b41b750c0spec:
> podManagementPolicy: Parallel
> replicas: 2
> revisionHistoryLimit: 10
> selector:
> matchLabels:
> app: heron
> topology: acking
> serviceName: acking
> template:
> metadata:
> annotations:
> prometheus.io/port: "8080"
> prometheus.io/scrape: "true"
> creationTimestamp: null
> labels:
> app: heron
> topology: acking
> spec:
> containers:
> - command:
> - sh
> - -c
> - './heron-core/bin/heron-downloader-config kubernetes &&
./heron-core/bin/heron-downloader
distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--7278444175656031685.tar.gz
. && SHARD_ID=$((${POD_NAME##*-} + 1)) && echo shardId=${SHARD_ID} &&
./heron-core/bin/heron-executor --topology-name=acking
--topology-id=acking17797dee-b1f1-4afa-8018-0054ffb2f988
--topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
--state-manager-root=/heron
--state-manager-config-file=./heron-conf/statemgr.yaml
--tmanager-binary=./heron-core/bin/heron-tmanager
--stmgr-binary=./heron-core/bin/heron-stmgr
--metrics-manager-classpath=./heron-core/lib/metricsmgr/*
--instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
--classpath=heron-api-examples.jar
--heron-internals-config-file=./heron-conf/heron_internals.yaml
--override-config-file=./heron-conf/override.yaml --componen
t-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
--pkg-type=jar --topology-binary-file=heron-api-examples.jar
--heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
--cluster=kubernetes --role=saad --environment=default
--instance-classpath=./heron-core/lib/instance/*
--metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
--scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
--python-instance-binary=./heron-core/bin/heron-python-instance
--cpp-instance-binary=./heron-core/bin/heron-cpp-instance
--metricscache-manager-classpath=./heron-core/lib/metricscachemgr/*
--metricscache-manager-mode=disabled --is-stateful=false
--checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
--stateful-config-file=./heron-conf/stateful.yaml
--checkpoint-manager-ram=1073741824 --health-
manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
--shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002
--tmanager-stats-port=6003 --shell-port=6004
--metrics-manager-port=6005 --scheduler-port=6006
--metricscache-manager-server-port=6007
--metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
> env:
> - name: HOST
> valueFrom:
> fieldRef:
> apiVersion: v1
> fieldPath: status.podIP
> - name: POD_NAME
> valueFrom:
> fieldRef:
> apiVersion: v1
> fieldPath: metadata.name
> image: apache/heron:testbuild
> imagePullPolicy: IfNotPresent
> name: executor
> ports:
> - containerPort: 6001
> name: server
> protocol: TCP
> - containerPort: 6004
> name: shell-port
> protocol: TCP
> - containerPort: 6002
> name: tmanager-ctl
> protocol: TCP
> - containerPort: 6005
> name: metrics-mgr
> protocol: TCP
> - containerPort: 6009
> name: ckptmgr
> protocol: TCP
> - containerPort: 6007
> name: metrics-cache-m
> protocol: TCP
> - containerPort: 6003
> name: tmanager-stats
> protocol: TCP
> - containerPort: 6008
> name: metrics-cache-s
> protocol: TCP
> - containerPort: 6006
> name: scheduler
> protocol: TCP
> resources:
> limits:
> cpu: "3"
> memory: 4Gi
> requests:
> cpu: "3"
> memory: 4Gi
> terminationMessagePath: /dev/termination-log
> terminationMessagePolicy: File
> dnsPolicy: ClusterFirst
> restartPolicy: Always
> schedulerName: default-scheduler
> securityContext: {}
> terminationGracePeriodSeconds: 0
> tolerations:
> - effect: NoExecute
> key: node.kubernetes.io/not-ready
> operator: Exists
> tolerationSeconds: 10
> - effect: NoExecute
> key: node.kubernetes.io/unreachable
> operator: Exists
> tolerationSeconds: 10
> updateStrategy:
> rollingUpdate:
> partition: 0
> type: RollingUpdatestatus:
> availableReplicas: 2
> collisionCount: 0
> currentReplicas: 2
> currentRevision: acking-executors-77ffc94579
> observedGeneration: 1
> readyReplicas: 2
> replicas: 2
> updateRevision: acking-executors-77ffc94579
> updatedReplicas: 2
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/incubator-heron/issues/3724#issuecomment-974696234>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AABMNKST27UOGZ4W6QDL33LUM7V5RANCNFSM5GVRYL7Q>
> .
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
