[
https://issues.apache.org/jira/browse/FLINK-36769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xingyuan cheng updated FLINK-36769:
-----------------------------------
Description:
Hi, community. Currently, in the batch verification scenario of our algorithm
data, we use pyflink and encounter low transmission efficiency caused by low
performance of pickle4-based encoding. After research, we decided to adopt
Apache fury, a serialization framework based on pickle5 encoding. The
implementation of fury in python will define the transmission buffer size in
the protocol for transmission to improve the performance of large data
transmission. To this end, I Prepared a draft pull request. What do friends in
the community think about this?
Pickle protocol 5 with out-of-band data: https://peps.python.org/pep-0574/
was:Hi, community. Currently, in the batch verification scenario of our
algorithm data, we use pyflink and encounter low transmission efficiency caused
by low performance of pickle4-based encoding. After research, we decided to
adopt Apache fury, a serialization framework based on pickle5 encoding. The
implementation of fury in python will define the transmission buffer size in
the protocol for transmission to improve the performance of large data
transmission. To this end, I Prepared a draft pull request. What do friends in
the community think about this?
> Suport fury Serializer for pyflink
> ----------------------------------
>
> Key: FLINK-36769
> URL: https://issues.apache.org/jira/browse/FLINK-36769
> Project: Flink
> Issue Type: New Feature
> Components: API / Python
> Affects Versions: 1.16.1
> Environment: flink 1.16.1
> Reporter: xingyuan cheng
> Priority: Major
>
> Hi, community. Currently, in the batch verification scenario of our algorithm
> data, we use pyflink and encounter low transmission efficiency caused by low
> performance of pickle4-based encoding. After research, we decided to adopt
> Apache fury, a serialization framework based on pickle5 encoding. The
> implementation of fury in python will define the transmission buffer size in
> the protocol for transmission to improve the performance of large data
> transmission. To this end, I Prepared a draft pull request. What do friends
> in the community think about this?
>
> Pickle protocol 5 with out-of-band data: https://peps.python.org/pep-0574/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
