karthik-0306 opened a new issue, #41404:
URL: https://github.com/apache/beam/issues/41404

   ### What happened?
   
   In a multi-model `KeyedModelHandler` (`KeyModelMapping`s), 
`_ModelHandlerManager.load` in `sdks/python/apache_beam/ml/inference/base.py` 
stores the key's tag in `_tag_map` (line 711) before the model is loaded (line 
728). If `load_model()` raises once, for example on a transient GCS error, the 
tag stays in `_tag_map` with no entry in `_proxy_map`, and that worker never 
recovers:
   
   1. Every later `load(key)` call finds the key in `_tag_map` and returns the 
stale tag. Nothing holds a reference to the model, so 
`KeyedModelHandler.run_inference` (line 1038) loads it again from scratch for 
every batch.
   2. When that key is evicted (`max_models_per_worker_hint`, line 719) or its 
model is updated (`update_model_handler`, line 778), `self._proxy_map[tag]` 
raises `KeyError: '<uuid>'` and the batch fails.
   
   Repro (calls the manager the same way `KeyedModelHandler.run_inference` 
does):
   
   ```python
   from apache_beam.ml.inference.base import ModelHandler, _ModelHandlerManager
   from apache_beam.utils import multi_process_shared
   
   LOADS = []
   
   class Model:
     def get(self):
       return 1
   
   class Handler(ModelHandler):
     def __init__(self, name, fail_first):
       self.name, self.fail_first = name, fail_first
     def load_model(self):
       if self.fail_first:
         self.fail_first = False
         raise IOError('transient error')
       LOADS.append(self.name)
       return Model()
     def run_inference(self, batch, model, inference_args=None):
       return batch
   
   def serve(mgr, key):
     stats = mgr.load(key)
     handle = multi_process_shared.MultiProcessShared(
         mgr._mh_map[key].load_model, tag=stats.model_tag)
     model = handle.acquire()
     try:
       return model.get()
     finally:
       handle.release(model)
   
   if __name__ == '__main__':
     mgr = _ModelHandlerManager({
         'ok': Handler('ok', False),
         'flaky': Handler('flaky', True),
         'new': Handler('new', False)
     })
     for _ in range(5):
       serve(mgr, 'ok')
     try:
       serve(mgr, 'flaky')
     except IOError:
       pass
     for _ in range(5):
       serve(mgr, 'flaky')
     print(LOADS.count('ok'), LOADS.count('flaky'))  # 1 5
   
     # Evicting 'flaky' (the least recently used key) fails.
     mgr.increment_max_models(2)
     serve(mgr, 'ok')
     serve(mgr, 'new')  # KeyError: '<uuid of the flaky tag>'
   ```
   
   Output: `load_model` runs once for 5 batches of `ok`, but 5 times for 5 
batches of `flaky`. The last call raises `KeyError` with a uuid.
   
   Expected: a failed `load_model` doesn't leave anything behind in `_tag_map`. 
The next batch retries the load once, and the model is then cached like any 
other key.
   
   I plan to remove the key from `_tag_map` when the load fails, and add a test.
   
   Version: master at e606c3b5ab (2.79.0.dev)
   
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [x] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Infrastructure
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Prism Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to