[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[
https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065851#comment-17065851
]
ASF subversion and git services commented on ARTEMIS-975:
-
Commit 4b49b3f3713001c94e5efeca017d8a9f3db499f3 in activemq-artemis's branch
refs/heads/master from Benjamin Graf
[ https://gitbox.apache.org/repos/asf?p=activemq-artemis.git;h=4b49b3f ]
ARTEMIS-975: Add transactional records to deletedRecords to check for committed
transactions that also hold references to
> Reading messages from page causes lost entries in db backend
>
>
> Key: ARTEMIS-975
> URL: https://issues.apache.org/jira/browse/ARTEMIS-975
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.0.0
>Reporter: Benjamin Graf
>Priority: Major
> Attachments: screenshot-1.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Consuming messages from pages with a db as persistence backend leaves commit
> records in message_table which doesn't get cleaned up. Seems that depaging
> does not add txids to internal map.
> SQL script for cleanup (no known side effect):
> {code}
> DELETE
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM
>(SELECT count(*) AS cnt,
>txid
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM MESSAGE_TABLE
> WHERE recordtype = 18)
> GROUP BY txid)AS m
> WHERE m.cnt=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[
https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064759#comment-17064759
]
Benjamin Graf commented on ARTEMIS-975:
---
Hi [~jbertram],
I finally found the root cause and fixed it.
Regards,
Benjamin
> Reading messages from page causes lost entries in db backend
>
>
> Key: ARTEMIS-975
> URL: https://issues.apache.org/jira/browse/ARTEMIS-975
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.0.0
>Reporter: Benjamin Graf
>Priority: Major
> Attachments: screenshot-1.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Consuming messages from pages with a db as persistence backend leaves commit
> records in message_table which doesn't get cleaned up. Seems that depaging
> does not add txids to internal map.
> SQL script for cleanup (no known side effect):
> {code}
> DELETE
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM
>(SELECT count(*) AS cnt,
>txid
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM MESSAGE_TABLE
> WHERE recordtype = 18)
> GROUP BY txid)AS m
> WHERE m.cnt=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[
https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063509#comment-17063509
]
Benjamin Graf commented on ARTEMIS-975:
---
The Latest findings through trace logging and heap dump analysis:
Between paging, depaging recordinfo entries are handled by several txids
causing some of them to stuck as transactionholder events not being completely
handled and removed from database backend.
Since a "lot" of transactionholder events are still addressed after consuming
all events it seems there is an issue how the whole thing is designed.
> Reading messages from page causes lost entries in db backend
>
>
> Key: ARTEMIS-975
> URL: https://issues.apache.org/jira/browse/ARTEMIS-975
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.0.0
>Reporter: Benjamin Graf
>Priority: Major
> Attachments: screenshot-1.png
>
>
> Consuming messages from pages with a db as persistence backend leaves commit
> records in message_table which doesn't get cleaned up. Seems that depaging
> does not add txids to internal map.
> SQL script for cleanup (no known side effect):
> {code}
> DELETE
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM
>(SELECT count(*) AS cnt,
>txid
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM MESSAGE_TABLE
> WHERE recordtype = 18)
> GROUP BY txid)AS m
> WHERE m.cnt=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[
https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055883#comment-17055883
]
Benjamin Graf commented on ARTEMIS-975:
---
New info. I can see that those txs that are still available in the database are
also still referenced in journal.transactions with non empty recordInfos list.
Actually don't know where they are coming from.
> Reading messages from page causes lost entries in db backend
>
>
> Key: ARTEMIS-975
> URL: https://issues.apache.org/jira/browse/ARTEMIS-975
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.0.0
>Reporter: Benjamin Graf
>Priority: Major
> Attachments: screenshot-1.png
>
>
> Consuming messages from pages with a db as persistence backend leaves commit
> records in message_table which doesn't get cleaned up. Seems that depaging
> does not add txids to internal map.
> SQL script for cleanup (no known side effect):
> {code}
> DELETE
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM
>(SELECT count(*) AS cnt,
>txid
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM MESSAGE_TABLE
> WHERE recordtype = 18)
> GROUP BY txid)AS m
> WHERE m.cnt=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[
https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015990#comment-17015990
]
Benjamin Graf commented on ARTEMIS-975:
---
[~jbertram]: Can still reproduce on 2.10.1 using mysql!
> Reading messages from page causes lost entries in db backend
>
>
> Key: ARTEMIS-975
> URL: https://issues.apache.org/jira/browse/ARTEMIS-975
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.0.0
>Reporter: Benjamin Graf
>Priority: Major
> Attachments: screenshot-1.png
>
>
> Consuming messages from pages with a db as persistence backend leaves commit
> records in message_table which doesn't get cleaned up. Seems that depaging
> does not add txids to internal map.
> SQL script for cleanup (no known side effect):
> {code}
> DELETE
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM
>(SELECT count(*) AS cnt,
>txid
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM MESSAGE_TABLE
> WHERE recordtype = 18)
> GROUP BY txid)AS m
> WHERE m.cnt=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[
https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015985#comment-17015985
]
Benjamin Graf commented on ARTEMIS-975:
---
[~jbertram]: I'm actually trying to reproduce. It's quite a long time ago.
Actually facing a new bug for using with mysql for query "append-to-file=SELECT
DATA FROM %s WHERE ID=? FOR UPDATE". Could be solved by adding ID (primary key
as requested by driver!) "append-to-file=SELECT DATA, *ID* FROM %s WHERE ID=?
FOR UPDATE"
> Reading messages from page causes lost entries in db backend
>
>
> Key: ARTEMIS-975
> URL: https://issues.apache.org/jira/browse/ARTEMIS-975
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.0.0
>Reporter: Benjamin Graf
>Priority: Major
> Attachments: screenshot-1.png
>
>
> Consuming messages from pages with a db as persistence backend leaves commit
> records in message_table which doesn't get cleaned up. Seems that depaging
> does not add txids to internal map.
> SQL script for cleanup (no known side effect):
> {code}
> DELETE
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM
>(SELECT count(*) AS cnt,
>txid
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM MESSAGE_TABLE
> WHERE recordtype = 18)
> GROUP BY txid)AS m
> WHERE m.cnt=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[
https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015361#comment-17015361
]
Justin Bertram commented on ARTEMIS-975:
Can you reproduce this with the latest release (i.e. 2.10.1 at this point)?
> Reading messages from page causes lost entries in db backend
>
>
> Key: ARTEMIS-975
> URL: https://issues.apache.org/jira/browse/ARTEMIS-975
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.0.0
>Reporter: Benjamin Graf
>Priority: Major
> Attachments: screenshot-1.png
>
>
> Consuming messages from pages with a db as persistence backend leaves commit
> records in message_table which doesn't get cleaned up. Seems that depaging
> does not add txids to internal map.
> SQL script for cleanup (no known side effect):
> {code}
> DELETE
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM
>(SELECT count(*) AS cnt,
>txid
> FROM MESSAGE_TABLE
> WHERE txid IN
> (SELECT txid
> FROM MESSAGE_TABLE
> WHERE recordtype = 18)
> GROUP BY txid)AS m
> WHERE m.cnt=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-975) Reading messages from page causes lost entries in db backend
[ https://issues.apache.org/jira/browse/ARTEMIS-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873049#comment-15873049 ] Benjamin Graf commented on ARTEMIS-975: --- I think I found the way the commit records get added (see screenshot-1.png) > Reading messages from page causes lost entries in db backend > > > Key: ARTEMIS-975 > URL: https://issues.apache.org/jira/browse/ARTEMIS-975 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.0.0 >Reporter: Benjamin Graf > Attachments: screenshot-1.png > > > Consuming messages from pages with a db as persistence backend leaves commit > records in message_table which doesn't get cleaned up. Seems that depaging > does not add txids to internal map. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
