Robert Bradshaw created BEAM-4030:

             Summary: Add CombineFn.compact, similar to Java
                 Key: BEAM-4030
             Project: Beam
          Issue Type: Bug
          Components: sdk-py-core
            Reporter: Robert Bradshaw
            Assignee: Ahmet Altay

Some CombineFns buffer elements in their add_inputs because a combining 
operation cost can be effectively amortized across many elements. However, this 
introduces the extra (possibly higher) cost of potentially serializing more 
expensive buffers through shuffle. We should add a CombineFn.compact(self, 
accumulator) method (defaulting to the identity) similar to what the Java SDK 
provides which is called when flushing an element from the PGBKCV table. 

This message was sent by Atlassian JIRA

Reply via email to