weiqingy commented on code in PR #950: URL: https://github.com/apache/flink-agents/pull/950#discussion_r3837388709
########## runtime/src/main/java/org/apache/flink/agents/runtime/memory/EventAttachmentUtils.java: ########## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.agents.runtime.memory; + +import org.apache.flink.agents.api.Event; +import org.apache.flink.agents.api.OutputEvent; +import org.apache.flink.agents.api.context.MemoryObject; +import org.apache.flink.agents.api.context.MemoryRef; +import org.apache.flink.agents.api.context.RunnerContext; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.Map; +import java.util.UUID; +import java.util.stream.Collectors; + +/** Stores event attachments in sensory memory while events cross action boundaries. */ +public final class EventAttachmentUtils { + + private static final String ATTACHMENT_ROOT = "__event_attachments__"; + + private EventAttachmentUtils() {} + + /** Stores concrete attachment values and replaces them with sensory-memory references. */ + public static void storeEventAttachments(Event event, RunnerContext context) throws Exception { + if (event.getAttachments().isEmpty()) { + return; + } + + if (OutputEvent.EVENT_TYPE.equals(event.getType())) { + String keys = + event.getAttachments().keySet().stream() + .sorted() + .collect(Collectors.joining(", ")); + throw new IllegalArgumentException( + "Output events cannot carry attachments: event_id=" + + event.getId() + + ", event_type=" + + event.getType() + + ", key=" + + keys); + } + + for (Map.Entry<String, Object> entry : event.getAttachments().entrySet()) { + String key = entry.getKey(); + Object value = entry.getValue(); + if (value instanceof MemoryRef) { + continue; + } + + MemoryRef reference = + context.getSensoryMemory().set(buildAttachmentPath(event.getId(), key), value); + + event.getAttachments().put(key, reference); + } + } + + /** Loads sensory-memory references in place before a Java action is invoked. */ + public static void loadEventAttachments(Event event, RunnerContext context) throws Exception { + for (Map.Entry<String, Object> entry : event.getAttachments().entrySet()) { + Object value = entry.getValue(); + if (!(value instanceof MemoryRef)) { + continue; + } + MemoryRef reference = (MemoryRef) value; + + MemoryObject attachment = context.getSensoryMemory().get(reference); + if (attachment == null) { + throw new IllegalStateException( + "Event attachment does not exist in sensory memory: " + + reference.getPath()); + } + event.getAttachments().put(entry.getKey(), attachment.getValue()); Review Comment: Yes, this closes it. The runtime event is only read now, so the one the durable store holds keeps its refs. One thing I'm unsure about: the copy is unconditional. `EventAttachmentUtils.java:97` copies before looking at anything, and every Java action goes through it (`JavaActionTask.java:61`), so an event with no attachments still pays for a full deep copy. `storeEventAttachments` bails out early on an empty map at `:51-53`, and the load side could do the same, since the only write is at `:111`, behind the `instanceof MemoryRef` check. Then again, maybe the uniform copy is the point. The javadoc at `:95` calls it an action-owned copy, and that isolation also keeps an action from mutating the live event's attributes, which `ActionStateUtil.generateUUIDForEvent` (`:65-67`) hashes into the durable state key. Which way were you thinking? Smaller one at `:41-42`: the serializer comes from a bare `new SerializerConfigImpl()`, so it has no registered serializers, while keyed state builds its own from the job config. A type registered through `pipeline.serialization-config` would apply when the event is persisted but not to this copy. Would building it from the operator's config and holding it as an instance field work? That drops the static `ThreadLocal` too. nit: `resolvesAttachmentsWithoutMutatingDurableEvent` checks the subclass and the attachments but nothing about `attributes`, so a copy that dropped them would still pass. `assertThat(((InputEvent) invokedEvent).getInput()).isEqualTo(1L)` would cover it, if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
