LuciferYang opened a new issue, #12790:
URL: https://github.com/apache/gluten/issues/12790
### Backend
VL (Velox). The code is in `gluten-core`; the caller that makes the growth
unbounded is in `gluten-arrow`.
### Bug description
`MemoryTargetUtil.toUniqueName` keeps a counter per name and never evicts:
```java
private static final Map<String, Integer> UNIQUE_NAME_LOOKUP = new
ConcurrentHashMap<>();
public static String toUniqueName(String name) {
int nextId = UNIQUE_NAME_LOOKUP.compute(
name, (s, integer) -> Optional.ofNullable(integer).map(id -> id +
1).orElse(0));
return String.format("%s.%d", name, nextId);
}
```
That would be bounded if names came from a fixed set, but they do not.
`NativePlanEvaluator.create` derives its runtime name from a JVM-global
`AtomicInteger`, so it produces `NativePlanEvaluator-0`,
`NativePlanEvaluator-1`, and so on, never repeating. That name reaches
`toUniqueName` through `Runtimes.contextInstance`, `NativeMemoryManager`,
`ReservationListeners`, and `TreeMemoryConsumer.Node`'s constructor. Each such
name appears exactly once, so its entry is written, never read again, and never
removed.
Both the executor and the driver accumulate entries.
On an executor it is one entry per whole-stage transformer per task, from
`VeloxIteratorApi` (two `NativePlanEvaluator.create` call sites).
On the driver it is one entry per native validation.
`VeloxValidatorApi#doNativeValidateWithFailureReason` and
`#doNativeValidateExpression` each build an evaluator inside
`TaskResources.runUnsafe` while planning, so a long-lived driver such as a
Thrift server or a notebook session validates candidate operators for every
query and accumulates faster per unit time than an executor, with nothing
released between queries.
There is a second, much less likely problem in the same method: the counter
is an `Integer`, so on a fixed name such as `Gluten.Tree` it would wrap to
`Integer.MIN_VALUE` after 2^31 memory targets.
### Gluten version
main (1.8.0-SNAPSHOT)
### Spark version
Version-agnostic (applies to spark-3.3 / 3.4 / 3.5 / 4.0 / 4.1).
### Spark configurations
None. Any query that reaches native validation on the driver, or runs a
whole-stage transformer on an executor, adds entries.
### System information
Not applicable. The affected code is
`gluten-core/src/main/java/org/apache/gluten/memory/memtarget/MemoryTargetUtil.java`,
and does not depend on OS or hardware.
### Relevant logs
No exception. The map is small per entry, so this shows up as slow heap
growth over the process lifetime rather than a failure.
### Fix direction
Use one `AtomicLong` for all names and return `name + "." +
SEQUENCE.getAndIncrement()`. The suffix only has to make the name unique,
because `TreeMemoryConsumer#newChild` keys siblings by name and throws on a
collision, and a shared sequence does that with no per-name state. A `long`
also removes the `Integer` wrap.
Suffixes stop being dense per name. The names that change most are the
high-cardinality ones that leaked: each appeared once, so its suffix was always
`.0`, and `NativePlanEvaluator-7.0` would read something like
`NativePlanEvaluator-7.913`. The fixed names were already climbing
JVM-globally, `Gluten.Tree` once per task, so only their scale changes. Nothing
depends on the number: it is a label for `SparkMemoryUtil`'s stats map and
pretty printer, it is not parsed, `memory.proto` carries no name field, and it
never crosses JNI, where only `backendName` is passed.
A bounded or weak cache would be worse than either option: evicting a live
target's entry lets the counter restart and hand a second sibling the same
name, turning `newChild`'s collision check from unreachable code into a task
failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
