jk-shah opened a new issue, #41347:
URL: https://github.com/apache/beam/issues/41347
### What happened?
`FeastFeatureStoreEnrichmentHandler`
(`sdks/python/apache_beam/transforms/enrichment_handlers/feast_feature_store.py`)
has three bugs:
1. **`get_cache_key` raises `KeyError` when `entity_row_fn` maps an input
field to a differently named Feast entity (lines `192-200`)**:
```python
def get_cache_key(self, request: beam.Row) -> str:
if self.entity_row_fn:
entity_dict = self.entity_row_fn(request)
entity_id = list(entity_dict.keys())[0]
else:
entity_id = self.entity_id
return 'entity_id: %s' % request._asdict()[entity_id]
```
When `entity_row_fn=lambda row: {'driver_id': row.raw_driver_id}` is used
(where `'driver_id'` is the Feast entity name, not a field on `request`),
`get_cache_key` looks up `request._asdict()['driver_id']` instead of
`entity_dict[entity_id]`, crashing with `KeyError: 'driver_id'` whenever
`Enrichment(...).with_redis_cache(...)` is enabled.
2. **`exception_level` parameter is dead code (lines `129-140` vs
`167-186`)**:
`__init__` documents `exception_level` ("to set the level when `None`
feature values are fetched from the online Feast store") and stores
`self._exception_level = exception_level` (line `140`), but
`self._exception_level` is never referenced anywhere in `__call__`
(`ExceptionLevel.RAISE` never raises and `ExceptionLevel.WARN` never logs on
missing entities).
3. **Leaked temporary YAML files in `/tmp` (lines `43-50`, `147`,
`188-190`)**:
`download_fs_yaml_file` creates `tempfile.NamedTemporaryFile(...,
delete=False)` in `__enter__`, and `__exit__` sets `self.store = None` without
unlinking the downloaded file.
Minimal reproducer for (1):
```python
from unittest import mock
import apache_beam as beam
from apache_beam.transforms.enrichment_handlers.feast_feature_store import (
FeastFeatureStoreEnrichmentHandler,
)
with mock.patch(
'apache_beam.transforms.enrichment_handlers.feast_feature_store.FileSystems.exists',
return_value=True):
h = FeastFeatureStoreEnrichmentHandler(
feature_store_yaml_path='/tmp/fs.yaml',
feature_names=['fv:score'],
entity_row_fn=lambda row: {'driver_id': row.raw_driver_id},
)
h.get_cache_key(beam.Row(raw_driver_id='d1'))
# Raises KeyError: 'driver_id'
```
### Cause
`get_cache_key` indexes `request._asdict()[entity_id]` instead of using
`entity_dict[entity_id]`; `__call__` omits checking `self._exception_level`
when fetched feature values are `None`; and `__exit__` does not unlink
`local_repo_path`.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [x] Component: Beam ML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]