[
https://issues.apache.org/jira/browse/ATLAS-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-2075:
-----------------------------------
Description:
*Background*
Messages produced by Hooks have potential to be larger than 1MB, which is the
size threshold imposed by Kafka.
Although, compressing the messages (see
[ATLAS-2064|https://issues.apache.org/jira/browse/ATLAS-2064]) alleviates the
the problem, it is not a complete solution. It is possible even for compressed
messages to exceed the size threshold.
*Solution*
If the compressed message produced exceeds the size threshold, split the
messages. Accumulate the message at consumer end.
Account for cases such as:
- Messages are not received in order they are produced.
- Atlas server is shutdown before it can consume all the split messages.
was:
*Background*
Messages produced by Hooks have potential to be larger than 1MB, which is the
size threshold imposed by Kafka.
Although, compressing the messages (see
[ATLAS-2064](https://issues.apache.org/jira/browse/ATLAS-2064)) alleviates the
the problem, it is not a complete solution. It is possible even for compressed
messages to exceed the size threshold.
*Solution*
If the compressed message produced exceeds the size threshold, split the
messages. Accumulate the message at consumer end.
Account for cases such as:
- Messages are not received in order they are produced.
- Atlas server is shutdown before it can consume all the split messages.
> Support Arbitrarily Large Size Messages from Hooks
> --------------------------------------------------
>
> Key: ATLAS-2075
> URL: https://issues.apache.org/jira/browse/ATLAS-2075
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
> Affects Versions: 0.8-incubating
> Reporter: Ashutosh Mestry
> Assignee: Ashutosh Mestry
> Fix For: trunk
>
> Attachments: ATLAS-2075-message-split-combine.patch
>
>
> *Background*
> Messages produced by Hooks have potential to be larger than 1MB, which is the
> size threshold imposed by Kafka.
> Although, compressing the messages (see
> [ATLAS-2064|https://issues.apache.org/jira/browse/ATLAS-2064]) alleviates the
> the problem, it is not a complete solution. It is possible even for
> compressed messages to exceed the size threshold.
> *Solution*
> If the compressed message produced exceeds the size threshold, split the
> messages. Accumulate the message at consumer end.
> Account for cases such as:
> - Messages are not received in order they are produced.
> - Atlas server is shutdown before it can consume all the split messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
