Hi Benoit,
I did have another try on this. Please have a look.
CREATE threadtable
CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid, username
text, mimeMessageId text, baseSubject text, PRIMARY KEY((username,
mimemessageid), messageid));
=> Partition key: (username, mimemessageid), clustering key: messageid.
To ease testing, I create a table with messageId and threadId type is int
instead.
Insert data
I will add:
-
2 related message and 1 unrelated message for user ‘quan’
-
1 message for user ‘benoit’ (which seem related to some messages of user
‘quan’)
// insert message1 data for ‘quan’
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (1, 1, 'quan', 'MimeMessageID1', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (1, 1, 'quan', 'MimeMessageID2', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (1, 1, 'quan', 'MimeMessageID3', 'baseSubject1');
// insert message2 data (related to message1) for ‘quan’
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (2, 1, 'quan', 'MimeMessageID1', 'baseSubject1');
// insert message3 data (not related to any message) for ‘quan’
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (3, 3, 'quan', 'MimeMessageID4', 'baseSubject2');
// insert message4 data (related to message1 but for ‘benoit’)
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (4, 4, 'benoit', 'MimeMessageID5', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (4, 4, 'benoit', 'MimeMessageID1', 'baseSubject1');
Select all data
The query for new messages
Because the subject can be null and Cassandra does not support selection by
null, so I will leave the base subject equal condition to the upper logic
layer.
Add a new related message
For example, there is a new message coming for user ‘quan’, with some
header fields:
-
SET MimeMessageIds (after combine values): {‘MimeMessageID2’,
‘MimeMessageID3’}
-
Base subject line (after stripping): “baseSubject1”
This message is supposed to be related to 2 other messages of ‘quan’.
We need to query one row related to this new message (if there is).
SELECT basesubject, threadId FROM threadtable WHERE username = 'quan' AND
mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3');
We will get these results and handle the rest subject condition at upper
level.
=> After that, this new message should have old threadId 1.
Add a new unrelated message
Supposed that we would add a message with unrelated mimemessage ids
SELECT basesubject, threadId FROM threadtable WHERE username = 'quan' AND
mimeMessageId IN (‘unrelatedMimeMessageID’);
As we can see, the identical mimeMessageID is not satisfied. So we will
immediately give this message a new threadId.
Insert new message data
After having a threadId, we need to insert new message data into the thread
table.
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (5, 1, 'quan', 'MimeMessageID2', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (5, 1, 'quan', 'MimeMessageID3', 'baseSubject1');
CREATE threadtable_lookup for deletion purpose
Supposed when we delete a message, we would need to delete that message’s
thread-related data in the threadtable also. I guess we just need messageId
to delete that message.
We would need to have another table similar to the threadtable but looked
up by messageId to get the needed params for deletion query.
CREATE TABLE ThreadTable_lookup (messageId timeuuid, username text,
mimeMessageIds set<text>, PRIMARY KEY(messageid));
To ease testing, I create a table with messageId’s type is int instead.
We will insert the data as same as the original table.
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (1, 'quan', {'MimeMessageID1', 'MimeMessageID2', 'MimeMessageID3'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (2, 'quan', {'MimeMessageID1'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (3, 'quan', {'MimeMessageID4'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (4, 'benoit', {'MimeMessageID5', 'MimeMessageID1'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (5, 'quan', {'MimeMessageID2', 'MimeMessageID3'});
SELECT * FROM threadtable_lookup;
Now we will do a query selection by messageId to get the needed username,
mimeMessageIds for original threadtable deletion.
Supposed we want to delete message 4.
SELECT username, mimemessageids FROM threadtable_lookup WHERE messageid = 4;
Then we will use these results to do a deletion query on threadtable.
The data of messageId 4 deleted.
Best regards,
Quan
Vào Th 3, 20 thg 7, 2021 vào lúc 18:39 btell...@apache.org <
btell...@apache.org> đã viết:
> Hello Quan,
>
> On 20/07/2021 17:24, Quan tran hong wrote:
> > [...]
> >
> > SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject
> =
> > 'baseSubject1' AND mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3')
> > LIMIT 1 ALLOW FILTERING;
> ALLOW FILTERING should not be used as it will result in a full scan and
> is thus a performance disaster.
>
> If you need it, this means you do not have the right table structure and
> likely should rework the CREATE TABLE statement.
> >
> > => This new message should have this threadId.
> > New unrelated message
> >
> > Assume that we do a query for a new unrelated message.
> >
> > SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject
> =
> > 'unrelatedBaseSubject' AND mimeMessageId IN ('MimeMessageID2',
> > 'MimeMessageID3') LIMIT 1 ALLOW FILTERING;
> >
> > => This new message should have a new threadId.
> > Insert new message data
> >
> > After having a threadId, we need to insert new message data into the
> thread
> > table.
> >
> > insert into ThreadTable (messageId, threadId, username, mimeMessageId,
> > baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
> > 'MimeMessageID2', 'baseSubject1');
> >
> > insert into ThreadTable (messageId, threadId, username, mimeMessageId,
> > baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
> > 'MimeMessageID3', 'baseSubject1');
> > Conclusion
> >
> > I think this data model complies with the needed request for the guessing
> > algorithm problem, but it looks like still maybe there is room for
> > improvement.
>
> What Cassandra request do we use to delete the data in there?
>
> >
> >
> > Best Regards,
> >
> > Quan
> >
> >
> >
> >
> >
> > Vào Th 2, 19 thg 7, 2021 vào lúc 18:23 btell...@apache.org <
> > btell...@apache.org> đã viết:
> >
> >> Hello Quan,
> >>
> >> On 19/07/2021 17:59, Quan tran hong wrote:
> >>> Hi,
> >>> I am starting to implement ThreadIdGuessingAlgorithm for the
> distributed
> >>> module. Because this is a breaking change and I am new to Cassandra
> also,
> >>> therefore I want to have some discussion with you about how to do this.
> >> As long as we introduce a new table there is no reason that it creates
> >> breaking change, but getting the format right will ease our life down
> >> the line.
> >>> For the ones who did not catch up with this work, please have a look at
> >>> JMAP Threads specs [1] and my work related to this [2].
> >>>
> >>> So my ideas on how to do this:
> >>> - Add a needed inputs Cassandra Table for guessing threadId algorithm.
> >>> Maybe a table likes:
> >>> CREATE TABLE ThreadRelatedTable (
> >>> threadId timeuuid,
> >>> messageId timeuuid,
> >>> mimeMessageIds SET<text>,
> >>> subject text,
> >>> PRIMARY KEY (mimeMessageIds, subject)
> >>> );
> >>> - Whenever we guess threadId for a new message, we access this table
> and
> >> do
> >>> the matching query to get related threadId(if there is) or decide new
> >>> message should have a new threadId.
> >>> - Whenever we save a new message, we save the thread-related data to
> this
> >>> table.
> >>>
> >>> This is my first come-up idea. Please express your thoughts about this.
> >> Collections are an advanced data modeling tool, that should be used with
> >> caution. I am not sure using it in a PRIMARY KEY is a good idea. I am
> >> not sure that does what you want (the full primary key should be
> >> specified to know which node hold the data.
> >>
> >> Also, once you found the message related to a thread you want to
> >> validate that the subject matches. This can be done on application side
> >> (James), and avoids complicated data model.
> >>
> >> I encourage you to validate your data model using a Cassandra in docker
> >> and executing CQL commands locally with CQLSH tool to simulate the
> >> queries you whish to do, and learn about your data model before even
> >> starting to implement it. IMO sharing CQL commands for creating the
> >> table, inserting data in it, and retrieving data from it would be a
> >> great follow up to this email.
> >>
> >> How would you populate the data of that table?
> >>
> >> Best regards,
> >>
> >> Benoit
> >>> Best regards,
> >>>
> >>> Quan
> >>>
> >>> [1] https://jmap.io/spec-mail.html#threads
> >>> [2] https://issues.apache.org/jira/browse/JAMES-3516
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> >> For additional commands, e-mail: server-dev-h...@james.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>
