[ https://issues.apache.org/jira/browse/ATLAS-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Mestry updated ATLAS-2075: ----------------------------------- Description: *Background* Messages produced by Hooks have potential to be larger than 1MB, which is the size threshold imposed by Kafka. Although, compressing the messages (see [ATLAS-2064|https://issues.apache.org/jira/browse/ATLAS-2064]) alleviates the the problem, it is not a complete solution. It is possible even for compressed messages to exceed the size threshold. *Solution* If the compressed message produced exceeds the size threshold, split the messages. Accumulate the message at consumer end. Account for cases such as: - Messages are not received in order they are produced. - Atlas server is shutdown before it can consume all the split messages. was: *Background* Messages produced by Hooks have potential to be larger than 1MB, which is the size threshold imposed by Kafka. Although, compressing the messages (see [ATLAS-2064](https://issues.apache.org/jira/browse/ATLAS-2064)) alleviates the the problem, it is not a complete solution. It is possible even for compressed messages to exceed the size threshold. *Solution* If the compressed message produced exceeds the size threshold, split the messages. Accumulate the message at consumer end. Account for cases such as: - Messages are not received in order they are produced. - Atlas server is shutdown before it can consume all the split messages. > Support Arbitrarily Large Size Messages from Hooks > -------------------------------------------------- > > Key: ATLAS-2075 > URL: https://issues.apache.org/jira/browse/ATLAS-2075 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Affects Versions: 0.8-incubating > Reporter: Ashutosh Mestry > Assignee: Ashutosh Mestry > Fix For: trunk > > Attachments: ATLAS-2075-message-split-combine.patch > > > *Background* > Messages produced by Hooks have potential to be larger than 1MB, which is the > size threshold imposed by Kafka. > Although, compressing the messages (see > [ATLAS-2064|https://issues.apache.org/jira/browse/ATLAS-2064]) alleviates the > the problem, it is not a complete solution. It is possible even for > compressed messages to exceed the size threshold. > *Solution* > If the compressed message produced exceeds the size threshold, split the > messages. Accumulate the message at consumer end. > Account for cases such as: > - Messages are not received in order they are produced. > - Atlas server is shutdown before it can consume all the split messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029)