dongjoon-hyun commented on code in PR #749:
URL:
https://github.com/apache/spark-kubernetes-operator/pull/749#discussion_r3561900027
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/context/SparkAppContext.java:
##########
@@ -70,6 +70,23 @@ public Optional<Pod> getDriverPod() {
.findAny();
}
+ /**
+ * Live lookup of the driver pod against the API server, bypassing the
informer cache.
+ *
+ * @return An Optional containing the driver Pod as known to the API server.
+ */
+ public Optional<Pod> getDriverPodFromApi() {
+ return josdkContext
+ .getClient()
+ .pods()
+ .inNamespace(sparkApplication.getMetadata().getNamespace())
+ .withLabels(driverLabels(sparkApplication))
+ .list()
+ .getItems()
+ .stream()
+ .findAny();
+ }
Review Comment:
This could be a regression due to the newly added excessive API calls.
In states like `DriverRequested`/`DriverStarted`, two or three
`AppResourceObserveStep`s run back-to-back in a single reconcile loop, so while
the informer cache is stale each step performs its own live API-server lookup
(and emits its own warn log). Since `SparkAppContext` is shared by all steps
within one reconcile loop, memoizing the lookup result — following the same
lazy-initialization pattern as `secondaryResourceSpec` — reduces this to a
single API call per loop. (If the lookup throws `KubernetesClientException`,
nothing is cached, so the next step naturally retries.)
```suggestion
/** Memoized live driver pod lookup - initialized upon the first attempt
in this reconcile. */
private Optional<Pod> driverPodFromApi;
/**
* Live lookup of the driver pod against the API server, bypassing the
informer cache. The
* result is memoized so that multiple reconcile steps within the same
reconcile loop share a
* single API call.
*
* @return An Optional containing the driver Pod as known to the API
server.
*/
public Optional<Pod> getDriverPodFromApi() {
synchronized (this) {
if (driverPodFromApi == null) {
driverPodFromApi =
josdkContext
.getClient()
.pods()
.inNamespace(sparkApplication.getMetadata().getNamespace())
.withLabels(driverLabels(sparkApplication))
.list()
.getItems()
.stream()
.findAny();
}
return driverPodFromApi;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
