[
https://issues.apache.org/jira/browse/FLINK-32650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748802#comment-17748802
]
李精卫 commented on FLINK-32650:
-----------------------------
[~libenchao] I simply use the JavaCodeSpliter in flink-table-code-splitter to
split the class from flink-protobuf codegen, and found that it cannot be used
directly. The main reason is that the decode/encode method in
GeneratedProtoToRow_xxx.java is static, so all the split methods and variables
should be static, but JavaCodeSpilter does not take this into consideration, so
the split code janino cannot be successfully compiled. Another point is that I
think it is a bit strange to let flink-protobuf rely on
flink-table-code-splitter. If I have my own splitter, I think it should be more
reasonable.
How to make JavaCodeSplitter support the splitting of static methods can be
solved in a new issue after refining the problem?
> Added the ability to split flink-protobuf codegen code
> ------------------------------------------------------
>
> Key: FLINK-32650
> URL: https://issues.apache.org/jira/browse/FLINK-32650
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.18.0
> Reporter: 李精卫
> Priority: Major
>
> h3. backgroud
> Flink serializes and deserializes protobuf format data by calling the decode
> or encode method in GeneratedProtoToRow_XXX.java generated by codegen to
> parse byte[] data into protobuf java objects. The size of the decode/encode
> codegen method body is strongly related to the number of defined fields in
> protobuf. When the number of fields exceeds a certain threshold and the
> compiled method body exceeds 8k, the decode/encode method will not be
> optimized by JIT, seriously affecting serialization or deserialization
> performance. Even if the compiled method body exceeds 64k, it will directly
> cause the task to fail to start.
> h3. solution
> So I proposed Codegen Splitter for protobuf parsing to split the
> encode/decode method to solve this problem.
> The specific idea is as follows. In the current decode/encode method, each
> field defined for the protobuf message is placed in the method body. In fact,
> there are no shared parameters between the fields, so multiple fields can be
> merged and parsed and written into the split method body. If the number of
> strings in the current method body exceeds the threshold, a split method will
> be generated, these fields will be parsed in the split method, and the split
> method will be called in the decode/encode method. By analogy, the
> decode/encode method including the split method is finally generated.
> after spilt code example
>
> {code:java}
> //代码占位符
> public static RowData
> decode(org.apache.flink.formats.protobuf.testproto.AdProfile.AdProfilePb
> message){
> RowData rowData=null;
> org.apache.flink.formats.protobuf.testproto.AdProfile.AdProfilePb message1242
> = message;
> GenericRowData rowData1242 = new GenericRowData(5);
> split2585(rowData1242, message1242);
> rowData = rowData1242;return rowData;
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
