jaehyeon-kim commented on code in PR #32187: URL: https://github.com/apache/beam/pull/32187#discussion_r1718882377
########## website/www/site/content/en/documentation/patterns/shared-class.md: ########## @@ -0,0 +1,187 @@ +--- +title: "Cache data using a shared object" +--- + +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Cache data using a shared object + +A cache is a software component that stores data so that future requests for that data can be served faster. Side inputs, stateful `DoFn` and calling an external service can be used for accessing a cache. The Python SDK provide another option in the shared module, which can be more efficient than side inputs in terms of the amount of memory required and simpler than stateful `DoFn`. + +The samples on this page show you how to use the Shared class of the [shared module](https://beam.apache.org/releases/pydoc/current/apache_beam.utils.shared.html) to enrich elements in both bounded and unbounded `PCollection`s. Two data sets are used in the samples - order and customer. The order records include customer IDs with which customer attributes are added by mapping the customer records. + +## Create a cache on a batch pipeline + +In this example, the customer cache is loaded as a dictionary in the `setup` method of the `EnrichOrderFn`, and it is used to add customer attributes to the order records. Note that a Shared object encapsulates a weak reference to a singleton instance of the shared resource. Therefore, we need to create a wrapper class since the Python dictionary does not support weak references. + +{{< highlight py >}} Review Comment: Let me update with the following snippet in a separate commit. ```python class EnrichOrderFn(beam.DoFn): def __init__(self): self._customers = {} self._shared_handle = shared.Shared() def setup(self): # setup is a good place to initialize transient in-memory resources. self._customer_lookup = self._shared_handle.acquire(self.load_customers) def load_customers(self): self._customers = expensive_remote_call_to_load_customers() return WeakRefDict(self._customers) def process(self, element): attr = self._customer_lookup.get(element["customer_id"], {}) yield {**element, **attr} ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
