potiuk commented on code in PR #45962:
URL: https://github.com/apache/airflow/pull/45962#discussion_r1931277176
##########
task_sdk/pyproject.toml:
##########
@@ -27,10 +27,11 @@ dependencies = [
"httpx>=0.27.0",
"jinja2>=3.1.4",
"methodtools>=0.4.7",
- "msgspec>=0.18.6",
"psutil>=6.1.0",
"structlog>=24.4.0",
"retryhttp>=1.2.0",
+ "orjson>=3.10.11",
Review Comment:
No, I am just curious - because I was just wondering if adding new
dependency is needed - every new dependency adds a security risk and
complexity, and with my Airflow Security beach Cleaning project, I am looking
closely at every new dependency added. Every single one increases supply chain
risk, so IMHO, we should become more and more careful and make sure that we add
new dependency when it's really needed.
That's mostly it. I am not sure if we can avoid, it by using what
FastAPI/Pydantic already support - i have not checked it in detail all I know
serializing back/forth is precisely what Pydantic does, so it would be
reasonable to assume we could serialize logs with it.
That's why I am asking question if that's been maybe considered - from what
I understand Pydantic can already serialize and deserialize arbitrary dicts
https://docs.pydantic.dev/2.3/usage/types/dicts_mapping/ , so my question is do
we need yet another library for that ?
And streaming is more of an optimization, - depending on the use cases for
that logging API , I guess we now send the logs in chunks, so it might not be
needed, but theorethically in the future we could just switch to an async
streaming API for that that could be more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]