[jira] Commented: (IMAP-193) UIDNEXT generation is a big performance killer and should get reworked
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913985#action_12913985 ] Eric Charles commented on IMAP-193: --- I had to reread a few times to realize the meaning of: *It is not intended to provide any guarantee that any message will have this unique identifier* So UIDNEXT is a mechanism to know if new messages are delivered in that mailbox. The only constraint is that it must be lower or equal than the UID that will be given to a message coming in the future. So the simple, and still difficult to find :) way you described maps the requirements. The JPA example is a simple one because you can use the auto_increment provided by the DB. For store not having that function such as maildir, jcr, nosql,...we should try to work with injection, meaning that an interface representing the UidNextProvider (whatever its name is). Some mailbox store such as JPA would be injected with a JPAUidNextProvider, others such as JCR,... could use a FileUidNextProvider for example, or whate. I suppose the injected component would be a singleton, would work with a map (mailboxId, lastUid) and it will have to be thread-safe. How do you see the impl ? UIDNEXT generation is a big performance killer and should get reworked -- Key: IMAP-193 URL: https://issues.apache.org/jira/browse/IMAP-193 Project: JAMES Imap Issue Type: Improvement Affects Versions: 0.1 Reporter: Norman Maurer Assignee: Norman Maurer Fix For: 0.2 In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc. This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value. From rfc: Note: The next unique identifier value is intended to provide a means for a client to determine whether any messages have been delivered to the mailbox since the previous time it checked this value. It is not intended to provide any guarantee that any message will have this unique identifier. A client can only assume, at the time that it obtains the next unique identifier value, that messages arriving after that time will have a UID greater than or equal to that value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big performance killer and should get reworked
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913993#action_12913993 ] Eric Charles commented on IMAP-193: --- oops, I was a bit confused. The *UidNextProvider* would be by default a MemoryUidNextProvider. For stores without auto_increment, I was also thinking to a *UidProvider* that would be injected. This would be also injected with the choice impl (could be FileUidProvider, JPAUidProvider,... or whatever an imaginative developer could think to :). UIDNEXT generation is a big performance killer and should get reworked -- Key: IMAP-193 URL: https://issues.apache.org/jira/browse/IMAP-193 Project: JAMES Imap Issue Type: Improvement Affects Versions: 0.1 Reporter: Norman Maurer Assignee: Norman Maurer Fix For: 0.2 In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc. This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value. From rfc: Note: The next unique identifier value is intended to provide a means for a client to determine whether any messages have been delivered to the mailbox since the previous time it checked this value. It is not intended to provide any guarantee that any message will have this unique identifier. A client can only assume, at the time that it obtains the next unique identifier value, that messages arriving after that time will have a UID greater than or equal to that value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big performance killer and should get reworked
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913998#action_12913998 ] Eric Charles commented on IMAP-193: --- back again. With the auto_increment to generate UID (for JPA), we may fall in the pitfall described on One drawback of this approach is described in IMAP4 Implementation Recommendations http://www.rfc-editor.org/rfc/rfc2683.txt Some server implementations have attempted to make UIDs unique across the entire server. This is inadvisable, in that it limits the life of UIDs unnecessarily. The UID is a 32-bit number and will run out in reasonably finite time if it's global across the server. If you assign UIDs sequentially in one mailbox, you will not have to start re-using them until you have had, at one time or another, 2**32 different messages in that mailbox. In the global case, you will have to reuse them once you have had, at one time or another, 2**32 different messages in the entire mail store. Suppose your server has around 8000 users registered (2**13). That gives an average of 2**19 UIDs per user. Suppose each user gets 32 messages (2**5) per day. That gives you 2**14 days (16000+ days = about 45 years) before you run out. That may seem like enough, but multiply the usage just a little (a lot of spam, a lot of mailing list subscriptions, more users) and you limit yourself too much. UIDNEXT generation is a big performance killer and should get reworked -- Key: IMAP-193 URL: https://issues.apache.org/jira/browse/IMAP-193 Project: JAMES Imap Issue Type: Improvement Affects Versions: 0.1 Reporter: Norman Maurer Assignee: Norman Maurer Fix For: 0.2 In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc. This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value. From rfc: Note: The next unique identifier value is intended to provide a means for a client to determine whether any messages have been delivered to the mailbox since the previous time it checked this value. It is not intended to provide any guarantee that any message will have this unique identifier. A client can only assume, at the time that it obtains the next unique identifier value, that messages arriving after that time will have a UID greater than or equal to that value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big performance killer and should get reworked
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12914033#action_12914033 ] Norman Maurer commented on IMAP-193: Yeah thats right... and we should not forgit this. But with auto_increment in the composite key we will get one sequence per mailbox. UIDNEXT generation is a big performance killer and should get reworked -- Key: IMAP-193 URL: https://issues.apache.org/jira/browse/IMAP-193 Project: JAMES Imap Issue Type: Improvement Affects Versions: 0.1 Reporter: Norman Maurer Assignee: Norman Maurer Fix For: 0.2 In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc. This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value. From rfc: Note: The next unique identifier value is intended to provide a means for a client to determine whether any messages have been delivered to the mailbox since the previous time it checked this value. It is not intended to provide any guarantee that any message will have this unique identifier. A client can only assume, at the time that it obtains the next unique identifier value, that messages arriving after that time will have a UID greater than or equal to that value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] Commented: (IMAP-193) UIDNEXT generation is a big performance killer and should get reworked
[ https://issues.apache.org/jira/browse/IMAP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12913684#action_12913684 ] Norman Maurer commented on IMAP-193: Ok, I think I found a good solution for this after thinking a bit about it today. Its so simple that it was really hard to get it ;) So here what we should do: * Let us remove the Mailbox.getLastUid() method and not make it persistent. * Simply return 1 when the Mailbox is selected first. After that on the next append to the mailbox we can set the nextUid value to returned uid + 1. * Register a MailboxListener to the Mailbox when selecting it to get notified by append operations to the mailbox (in concurrent sessions) and set it when a append happen. Then the implementations are free to use whatever method/pattern they like to generate the uid on append. For example in JPA we could just use a auto_increment field. This would allow us to get rid of locking the row and so give better performance. For sure we need to update the unit tests to allow this kind of generation of uid when append. I think this is a way better then what we do now and is fully rfc conform (See RFC snipped in previous comment) WDYT ? UIDNEXT generation is a big performance killer and should get reworked -- Key: IMAP-193 URL: https://issues.apache.org/jira/browse/IMAP-193 Project: JAMES Imap Issue Type: Improvement Reporter: Norman Maurer Assignee: Norman Maurer In the store api we generate the uid for a new message and guaranteer that the Mailbox.getLastUid() method will return this value after it is stored. For this we save the message and the mailbox and use for example a row lock (in jpa land) for this to get rid of ghost-reads etc. This is a big performance killer and is not needed at all. In the rfc its clearly stated that the used uid for the next saved message must just be equal or bigger then the UIDNEXT value. From rfc: Note: The next unique identifier value is intended to provide a means for a client to determine whether any messages have been delivered to the mailbox since the previous time it checked this value. It is not intended to provide any guarantee that any message will have this unique identifier. A client can only assume, at the time that it obtains the next unique identifier value, that messages arriving after that time will have a UID greater than or equal to that value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
