wenjin272 commented on code in PR #466: URL: https://github.com/apache/flink-agents/pull/466#discussion_r2724862661
########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,262 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleand after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrival**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs: + +{{< tabs "MemoryObject" >}} + +{{< tab "Python" >}} +```python +memory_object.set("key1", value1) +memory_object.set("key2", value2) +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +memoryObject.set("key1", value1) +memoryObject.set("key2", value2) +``` +{{< /tab >}} + +{{< /tabs >}} + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # write values to memory + memory.set("primitive", 123) + memory.set("collection", [1, 2, 3]) + memory.set("object", ChatMessage(role=MessageRole.USER, content="test")) + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: ChatMessage = memory.get("object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // write values to memory + memory.set("primitive", 123); + memory.set("collection", List.of(1, 2, 3)); + memory.set("object", new ChatMessage(MessageRole.USER, "test")); + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + ChatMessage value3 = (ChatMessage) memory.get("object").getValue(); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +#### Nested Object + +Nested objects can be read and write in two ways: +* Use `new_object` to manually create nested field. +* Use dot-separated path as key to crated nested field. + +{{< tabs "Nested Object Access" >}} + +{{< tab "Python" >}} +```python +# Use new_object +user: MemoryObject = memory.new_object("user") +user.set("name", "john") +user.set("age", 13) + +user: MemoryObject = memory.get_object("user") +name: str = user.get("name") +age: int = user.get("age") + +# Use dot-separated path +memory.set("user.name", "jhon") +memory.set("user.age", 13) + +name: str = memory.get("user.name") +age: int = memory.get("user.age") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +// Use new_object +MemoryObject user = memory.newObject("user", true); +user.set("name", "john"); +user.set("age", 13); + +user = sensoryMemory.get("user"); +String name = (String) user.get("name").getValue(); +int age = (int) user.get("age").getValue(); + +// Use dot-separated path +sensoryMemory.set("user.name", "john"); +sensoryMemory.set("user.age", 13); + +name = (String) sensoryMemory.get("user.name").getValue(); +age = (int) sensoryMemory.get("user.age").getValue(); +``` +{{< /tab >}} + +{{< /tabs >}} + +### Memory Reference + +`MemoryRef` is a reference of the objects stored in memory. The `set` method of sensory or short-term memory will return a `MemoryRef`. + +#### When to use +`MemoryRef` is useful for passing data across multiple actions via memory. We recommend user to always use `MemoryRef` in events rather than original data. It can bring follow benefit: +* **Reduce the event payload size**: The size of `MemoryRef` is usually typically smaller than that of the original data. For event will be used not only for orchestration, but also for observability and fault tolerance. The reduction of event size can help reduce some cost. +* **Redundant data copy & SerDe**: To argue this point, we first need to clarify what copy and SerDe occur when using original data versus memory reference, respectively. + * Use `MemoryRef`: + * Write/read original data to/from memory will cause kryo SerDe. + * Use original data + * For python actions, write/read data to/from event will cause pickle SerDe. + * For java actions, write/read to/from event won't cause any SerDe. + * Json SerDe for event log and per-action state. + + Thus, using `MemoryRef` can avoid the data copy and SerDe for event log and per-action state. + +{{< tabs "Memory Reference" >}} + +{{< tab "Python" >}} +```python +@staticmethod +def first_action(event: Event, ctx: RunnerContext): + ... + sensory_memory = ctx.sensory_memory + + data_ref = sensory_memory.set(data_path, data_to_store) + ctx.send_event(MyEvent(value=data_ref)) + ... + +@action(MyEvent) +@staticmethod +def second_action(event: Event, ctx: RunnerContext): + ... + sensory_memory = ctx.sensory_memory + + content_ref: MemoryRef = event.value + processed_data: ProcessedData = sensory_memory.get(content_ref) Review Comment: Ok, I will create a follow issue for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
