[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843173#comment-17843173 ] Ahmed Hamdy commented on FLINK-21373: - [~martijnvisser] Thanks for assigning it to me, I have already opened a [PR|[GitHub Pull Request #29|https://github.com/apache/flink-connector-rabbitmq/pull/29]], your review is appereciated > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: Ahmed Hamdy >Priority: Minor > Labels: auto-unassigned, pull-request-available > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843160#comment-17843160 ] Ahmed Hamdy commented on FLINK-21373: - [~martijnvisser] I would love to work on this to close it out! > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Minor > Labels: auto-unassigned, pull-request-available > Fix For: 1.12.0 > > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454587#comment-17454587 ] ZhuoYu Chen commented on FLINK-21373: - [~westphal-jan] Hello, I would like to know how is the progress? > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Minor > Labels: auto-unassigned > Fix For: 1.12.0 > > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333611#comment-17333611 ] Flink Jira Bot commented on FLINK-21373: This issue was marked "stale-assigned" and has not received an update in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: Jan Westphal >Priority: Minor > Labels: stale-assigned > Fix For: 1.12.0 > > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322841#comment-17322841 ] Flink Jira Bot commented on FLINK-21373: This issue is assigned but has not received an update in 7 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: Jan Westphal >Priority: Minor > Labels: stale-assigned > Fix For: 1.12.0 > > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285775#comment-17285775 ] Arvid Heise commented on FLINK-21373: - [~Wosinsan] would you like to review a potential PR? We are also happy to take contributions on other connectors. > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: Jan Westphal >Priority: Minor > Fix For: 1.12.0 > > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285122#comment-17285122 ] Jan Westphal commented on FLINK-21373: -- We are already working on it in the context of a student project and are about to finish this issue. Thanks for your interest though. > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Minor > Fix For: 1.12.0 > > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21373) Port RabbitMQ Sink to FLIP-143 API
[ https://issues.apache.org/jira/browse/FLINK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284983#comment-17284983 ] Dominik Wosiński commented on FLINK-21373: -- Can I tackle this one ?? > Port RabbitMQ Sink to FLIP-143 API > -- > > Key: FLINK-21373 > URL: https://issues.apache.org/jira/browse/FLINK-21373 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Minor > Fix For: 1.12.0 > > > *Structure* > The unified Sink API provides a Writer, a Committer and a GlobalCommitter. > Right now we don’t see the need to use the Committer and GlobalCommitter as > the Writer is sufficient to hold up to the consistencies. Since we are in the > need of asynchronous RabbitMQ callbacks to know whether or not a message was > published successfully and have to store unacknowledged messages in the > checkpoint, there would be a large bidirectional communication and state > exchange overhead between the Writer and the Committer. > *At-most-once* > The Writer receives a message from Flink and simply publishes it to RabbitMQ. > The current RabbitMQ Sink only provides this mode. > *At-least-once* > The objective here is, to receive an acknowledgement by RabbitMQ for > published messages. Therefore, before publishing a message, we store the > message in a Map with the sequence number as its key. If the message is > acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive > an acknowledgement for a certain amount of time (or a RabbitMQ specific so > called negative acknowledgement) we will try to resend the message when > doing a checkpoint. > *Exactly-once* > On checkpointing we send all messages by Flink in transaction mode to > RabbitMQ. This way, all the messages get sent or are rolled back on failure. > All messages that are not sent successfully are written to the checkpoint and > are tried to be sent with the next checkpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
