bolkedebruin commented on PR #36319:
URL: https://github.com/apache/airflow/pull/36319#issuecomment-1866055628
Mmm yes. Looking a the thread on Slack the issue is the size of the list, so
it becomes a bit hard to do a
```python
def get_files(p: ObjectStoragePath) -> list[ObjectStoragePath]:
return [x for x in p.iterdir() if x.is_file()]
```
Am I right?
For this particular case it might be better to implement something like a
`XComManager` that uses `ThresholdBufferedStream` that write out to an
ObjectStoragePath in, preferably, Arrow format when crossing a certain size of
the object instead of adding it to the database.
So solving it at the XCom layer instead of doing it here. Because, as
@uranusjr mentioned it feels a bit wrong at this layer. The additional benefit
of doing it at the XCom layer is that it will benefit all values that go beyond
a certain size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]