Wes McKinney created ARROW-5767: ----------------------------------- Summary: [Format] Permit dictionary replacements in IPC protocol Key: ARROW-5767 URL: https://issues.apache.org/jira/browse/ARROW-5767 Project: Apache Arrow Issue Type: New Feature Components: Format Reporter: Wes McKinney Fix For: 1.0.0
We permit dictionaries to grow using the {{isDelta}} property in the IPC protocol. I think it should be allowed for the same dictionary ID to appear in an IPC protocol stream but with {{isDelta=false}}. This would indicate that the dictionary in that message is to replace any prior-observed ones in subsequent record batches. For example, we might have dictionary batches in a stream: {code} id: 0 isDelta: false values: [a, b, c] id: 0 isDelta: true values [d] id 0 isDelta: false values [c, a, b] {code} Such data could easily be produced by a stream producer that is creating dictionaries in different execution threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)