Re: Cyrus 3.0.3 XAPPLEPUSH

[Arnaldo Viegas de Lima]

There is not much that can be done for the error handling, as it’s a script 
called from a daemon. Stdout and stderr are not captured, so you need to resort 
to sending things to syslog.

I really haven’t worked with any other notification service, so I can’t comment 
on that note.

As for getting this to work on a larger scale there are basically 3 
considerations:

- Port the script code to a more efficient implementation (C or C++), mostly as 
a replacement to notifyd. Actually notifyd can be easily modified to play this 
role directly.
- Decide if MySQL is the way to go for your installation. Maybe using Cyrus 
twoskip? I used MySQL because it was available and simple to use.
- Sort out any issues with the certificate/key. I do not know how the licensing 
for using it work. My guess is that for a small shop they will not care if you 
extracted the certificate from macOS X, but running it for a few thousand users 
is another story. Having support for iOS Push Mail adds value to Apple’s 
products, so it may be simpler than I think and it’s just another push service: 
they have tons.

Arnaldo.

> On Aug 24, 2017, at 12:14 PM, Sebastian Hagedorn  
> wrote:
> 
> Hi Arnaldo,
> 
> OK, got it! I noticed that the error handling isn't really fleshed out ;-) I 
> could see that the script got the registration message, but the tables were 
> stil empty. Turns out I had to install php-mysql ... :-) Seems obvious in 
> hindsight, but there was no error message. I don't usually use PHP, as I'm 
> more of a Perl/Python guy, so I don't really know how to improve that part.
> 
> FWIW, I copied the handler for MessageNew to MessageAppend, because that's 
> the only way I can get messages on my test server right now, and it worked 
> perfectly.
> 
> What would it take to make this work on a larger scale? I haven't worked with 
> notifications at all until now, so I don't know how other mechanisms are 
> implemented.
> 
> Thanks, Sebastian
> 
> --On 24. August 2017 um 10:51:37 -0300 Arnaldo Viegas de Lima 
>  wrote:
> 
>> You don’t need to do anything with the tables. The script will populate
>> the tables during the device registration event.
> 
> -- 
>   .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
>.:.Regionales Rechenzentrum (RRZK).:.
>  .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.4.12 vs 3.0.2 different protocol behaviour

[Gabriele Bulfon]

:) I noticed I pasted them by mistake, it's a test user on a test server, but I 
changed its password anyway, already ;)
Thanks!
Sonicle S.r.l.
:
http://www.sonicle.com
Music:
http://www.gabrielebulfon.com
Quantum Mechanics :
http://www.cdbaby.com/cd/gabrielebulfon
--
From: Simon Matter
To: Gabriele Bulfon
Cc: Merlin Hartley
info-cyrus@lists.andrew.cmu.edu
Date: August 24, 2017 5:26:24 PM CEST
Subject: Re: 2.4.12 vs 3.0.2 different protocol behaviour
I looked deeper into the problem.
Looks like the command:
. LIST "" "Other Users"
responds abnormally, until I run :
. LIST "" "Other Users/%"
that actually list the shared folders.
Here's a trace of the imap protocol:
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR]
www Cyrus IMAP 3.0.2 server ready
. login gabriele.bul...@sonicle.com Nrdstg88!
I hope the login credentials are only for your test here?
Simon

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.4.12 vs 3.0.2 different protocol behaviour

[Simon Matter]

> I looked deeper into the problem.
> Looks like the command:
> . LIST "" "Other Users"
> responds abnormally, until I run :
> . LIST "" "Other Users/%"
> that actually list the shared folders.
> Here's a trace of the imap protocol:
> * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR]
> www Cyrus IMAP 3.0.2 server ready
> . login gabriele.bul...@sonicle.com Nrdstg88!

I hope the login credentials are only for your test here?

Simon


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 3.0.3 XAPPLEPUSH

[Sebastian Hagedorn]

Hi Arnaldo,

OK, got it! I noticed that the error handling isn't really fleshed out ;-) 
I could see that the script got the registration message, but the tables 
were stil empty. Turns out I had to install php-mysql ... :-) Seems obvious 
in hindsight, but there was no error message. I don't usually use PHP, as 
I'm more of a Perl/Python guy, so I don't really know how to improve that 
part.


FWIW, I copied the handler for MessageNew to MessageAppend, because that's 
the only way I can get messages on my test server right now, and it worked 
perfectly.


What would it take to make this work on a larger scale? I haven't worked 
with notifications at all until now, so I don't know how other mechanisms 
are implemented.


Thanks, Sebastian

--On 24. August 2017 um 10:51:37 -0300 Arnaldo Viegas de Lima 
 wrote:



You don’t need to do anything with the tables. The script will populate
the tables during the device registration event.


--
   .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
.:.Regionales Rechenzentrum (RRZK).:.
  .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.

pgpmbgzT9mAAx.pgp
Description: PGP signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 3.0.3 XAPPLEPUSH

[Arnaldo Viegas de Lima]

You don’t need to do anything with the tables. The script will populate the 
tables during the device registration event.

Here is my understanding on how it works, as it’s not documented:

Part 1: Registration

a. If Cyrus is properly configured, it will advertise the XAPPLEPUSH extension
b. The iOS device will recognize the availability and send a XAPPLEPUSH 
command, that has the following information:
APS Version: 1 or 2 
APS Account Id: UUID of the the account on the phone - allow for 
multiple accounts on the same device
APS Device Token: Device identification for the push message
APS: Subtopic: fixed string com.apple.mobilemail (per Apple’s source 
code and that’s what I’ve seen so far)
Mailbox: INBOX for version 1 and mailbox list for version 2
c. Cyrus will generate an ApplePushService event and relay all the parameters 
above to the event handler, along with the user name.
d. The event handler, in the script, add/update the registration for what I 
call a push target: device_token+account_id. Each target has a list of 
subscribed mailboxes (user+mailbox).
e. Whenever iOS sends a XAPPLEPUSH, the script updates the timestamp of the 
registration. This timestamp can be used to delete the registration records, 
for devices that do not renew the subscription after a while. I will add an 
option in the script to do it, but the important is that the timestamp is there.
f. Cyrus will return the “aps_topic” information, that comes from imapd.conf. 
This will be used by the iOS device to authenticate the received push 
notifications.

Notes on the registration:

1.  I’m trying to figure out how frequent devices “register”, but I’m still to 
find a pattern here. So far I’ve noticed that devices will renew the 
registration at least once a day. If that confirms to be a pattern, I would say 
that it’s safe to remove a device that fails to register after 10 days (allow 
for that cruise vacation without internet access - well even cruises have 
internet now).
2. Multiple mailboxes id quite useless (at least I couldn’t find a way to make 
Mail.App react to pushes for it. We use Cyrus delivery to mailbox option (both 
with filters and direct addressing, aka plussed users). I can register and send 
a push, but nothing happens on the device.


Part 1: Events

a. Whenever an event that is being monitored (event groups) occur, the external 
event handler (the script) is called, with the event’s message on stdin. This 
is the normal Cyrus behavior for externally handling events from notifyd. The 
script will decide what events to act upon. Right now I’m processing:
- MessageNew- This is 
obvious
- MessageRead (INBOX only)  - This will catch when 
a user reads a message on another device (iOS or otherwise)
- MessageTrash (INBOX only) - Same as above, as the 
user may trash a message without reading it (I do it all the time)
- MessageMove (vnd.cmu.MessageMove) - User moves message out of 
INBOX before reading
- FlagsClear (for \Seen)- 
Reverse of MessageRead 
b. Based on the event type, retrieve the user and mailbox information.
c. Query the database for all targets (device_token+account_id) that have 
registered to receive push notification for that user+mailbox combination
d. Send the push request to Apple.
- The push information payload contains  the Device_Token, Account_Id. 
- The certificate used for the push contains the aps_topic information 
- These are the 3 pieces that are needed for the iOS Mail.App to 
properly react. No mailbox information is sent. 
e. Mail.App on the iOS device will react, chime (if new INBOX item) and update 
the badge (increase or decrease).
 
Notes on Events:

1. Badge will only count unseen messages in INBOX (actually the sum of the 
INBOX of all defined accounts).
2. You will get a chime on badge increase but not on decrease. And only if the 
message is NEW (clearing \Seen does not causes a chime). That’s a nice touch of 
Apple’s implementation.
1. Delivery to mailboxes other than INBOX will not affect the badge. Actually I 
can’t get much out of it. The only thing I noticed on Mail.App is that if you 
are looking at the folder, it gets updated (but I’m not sure it’s reaction to 
the push).


I’ve tried other payloads, without success. Since there is no documentation, 
it’s hard to do. But I’m extremely satisfied with the set of events being 
monitored and they way iOS reacts to it. 

I would love to see comments from other people that have used it, specially 
about the events and payloads. 

And a final comment: the script approach is only good for a small shop (we have 
only 10 users, but with huge mailbox trees). 

Arnaldo.
Ps.I have updated the script since I’ve posted. The new version includes 
MessageRead/Unread (clear \Seen Flag) and sends more 

Re: 2.4.12 vs 3.0.2 different protocol behaviour

[Gabriele Bulfon]

I looked deeper into the problem.
Looks like the command:
. LIST "" "Other Users"
responds abnormally, until I run :
. LIST "" "Other Users/%"
that actually list the shared folders.
Here's a trace of the imap protocol:
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR] www 
Cyrus IMAP 3.0.2 server ready
. login gabriele.bul...@sonicle.com Nrdstg88!
. OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA 
MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN 
MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ 
SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS 
ANNOTATEMORE ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS 
LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE 
CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION URLAUTH URLAUTH=BINARY 
LOGINDISABLED XCONVERSATIONS COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE 
X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] User logged in 
SESSIONID=
. LIST "" "Other Users"
. OK Completed (0.010 secs)
. LIST "" "Other Users/%"
* LIST (\HasChildren) "/" "Other Users/amministrazione"
* LIST (\HasChildren) "/" "Other Users/dev-team"
* LIST (\HasChildren) "/" "Other Users/newsletter"
* LIST (\HasChildren) "/" "Other Users/pec"
* LIST (\HasChildren) "/" "Other Users/pectest"
* LIST (\HasChildren) "/" "Other Users/sonicle"
* LIST (\HasChildren) "/" "Other Users/spamadm"
* LIST (\HasChildren) "/" "Other Users/supporto"
. OK Completed (0.030 secs 162 calls)
. LIST "" "Other Users"
* LIST (\Noselect \HasChildren) "/" "Other Users"
. OK Completed (0.020 secs 162 calls)
As you can see the first check on presence of "Other Users" returns nothing.
While the last one, after listing subfolders, returns the correct answer.
JavaMail internally runs the check before looking for subfolders when trying to 
list a folder, and this fails only on "Other Users" because of the problem 
above.
Any idea what's happening?
Sonicle S.r.l.
:
http://www.sonicle.com
Music:
http://www.gabrielebulfon.com
Quantum Mechanics :
http://www.cdbaby.com/cd/gabrielebulfon
From:
Gabriele Bulfon
To:
Merlin Hartley
Cc:
info-cyrus@lists.andrew.cmu.edu
Date:
August 23, 2017 5:34:11 PM CEST
Subject:
Re: 2.4.12 vs 3.0.2 different protocol behaviour
In the reconstruct log, I have all the:
Repacked user/blabla/dra...@sonicle.com to version 13
for all the mailbox trees.
Obviously "Other Users" is not in reconstruct, as it is a virtual folder coming 
out of the incoming sharings.
ACLs looks same with cyradm for shared folders:
localhostlam user/soni...@sonicle.com
gabriele.bul...@sonicle.com lrswipkxtecda
soni...@sonicle.com lrswipkxtecda
localhostlam user/soni...@sonicle.com
gabriele.bul...@sonicle.com lrswipkxtecda
soni...@sonicle.com lrswipkxtecda
Sonicle S.r.l.
:
http://www.sonicle.com
Music:
http://www.gabrielebulfon.com
Quantum Mechanics :
http://www.cdbaby.com/cd/gabrielebulfon
From:
Merlin Hartley
To:
Gabriele Bulfon
Cc:
info-cyrus@lists.andrew.cmu.edu
Date:
August 23, 2017 5:16:22 PM CEST
Subject:
Re: 2.4.12 vs 3.0.2 different protocol behaviour
Just a quickie: Are you sure these mailboxes migrated properly?
M
--
Merlin Hartley
Computer Officer
MRC Mitochondrial Biology Unit
Cambridge, CB2 0XY
United Kingdom
On 23 Aug 2017, at 15:55, Gabriele Bulfon
gbul...@sonicle.com
wrote:
I would add that both systems returns the same list of folders with :
. LIST "" %
including the "Other Users" :
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" Archive
...
* LIST (\Noselect \HasChildren) "/" "Other Users"
but then the java code just tries a "folder.list()" on each of them, to see for 
subfolders.
Here the javamail code, before issuing the list command, tries the 
"checkExistance" using :
. LIST "" "Other Users"
and 3.0.2 returns an empty result.
Also trying by hand :
. LIST "Other Users" *
returns all the subtree on 2.4.12, while it's empty on 3.0.2.
Any idea?
Gabriele
Sonicle S.r.l.
:
http://www.sonicle.com
Music:
http://www.gabrielebulfon.com
Quantum Mechanics :
http://www.cdbaby.com/cd/gabrielebulfon
From:
Gabriele Bulfon
gbul...@sonicle.com
To:
info-cyrus@lists.andrew.cmu.edu
Date:
August 23, 2017 4:41:36 PM CEST
Subject:
2.4.12 vs 3.0.2 different protocol behaviour
Hi,
I found a problem while scanning the imap tree through our java web app, 
behaving differently from a 2.4.12 to a 3.0.2 upgraded.
Looking at the protocol I find these differences:
==2.4.12==
. LIST "" "Other Users"
* LIST (\Noselect \HasChildren) "/" "Other Users"
. OK Completed (0.000 secs 192 calls)
==3.0.2==
. LIST "" "Other Users"
. OK Completed (0.000 secs)
So, what happens is that while the java code is checking existence of a folder 
in the tree, on 3.0.2 it receives an error.
Systems are same, replicated, but one upgraded to 3.0.2, so imap base and 
configuration is the same, but for the 3.0.2 trimmings.
Is there any setting for compatibility in this case?
Thanx!
Gabriele
Sonicle S.r.l.
:

Re: Cyrus 3.0.3 XAPPLEPUSH

[Sebastian Hagedorn]

Hi Arnaldo,

thank you very much! This has been very enlightening. I think I understand 
most of it and have already set up a testing environment with a certificate 
from our OS X Server. What I have not yet understood completely is how you 
populate the database. Where do I find the "APNS Account Id" and the "APNS 
Device Token" for the devices table, and what do I put in the mailboxes 
table?


Thanks,
Sebastian

--On 23. August 2017 um 08:27:04 -0300 Arnaldo Viegas de Lima 
 wrote:



A few notes:

- You need a valid push certificate. You can get one from macOS Server,
but I do not know the limitations on the usage. My shop is really small,
so I think it’s not a problem.

- There is close to nothing documenting how this work, so much of it is
based on looking at the source code (Cyrus), from info on a similar
Dovecot plugin and on the actual Apple source code (based on Dovecot).
Also a lot of debugging and trial and error.

- Contrary to the existing (and minimal) reference in the docs, there is
no “—enable-apple-push-service” configure flag. The code in imapd
is so simple and harmless that it’s always compiled.

- XAPPLEPUSH requires TLS, so if TLS is not configured it will not be
advertised.

- XAPPLEPUSH is only advertised with the proper notifications settings
(see bellow).

- I used MySQL because it’s already running on the same server as Cyrus
and I have a nice set of functions to make scripts like this a breeze.
For the same reasons, it’s in PHP.


This is what needs to be added to imapd.conf:

# Apple Push Service Events
event_notifier: external
event_groups: applepushservice message flags
event_extra_params: timestamp vnd.cmu.unseenMessages
notify_external: /usr/local/bin/notifyd_apns
aps_topic: com.apple.mail.XServer.xxx

aps_topic is the common name take from the certificate. It’s sent to
the mobile device so that it will match the source of the push
notification when it arrives.


The MySQL database configuration is also attached.

For the certificate you need to enable mail services and export the
“mobilemail” certificate and key (as pkcs12) from the keychain. Then
use openssl to remove key and certificate as well as to unencrypt the
key. Combine both in a single PEM file.

Enjoy!


pgpMuMPum_mtr.pgp
Description: PGP signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus