Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-02-09 Thread Rahul Amaram 



On Sunday 31 January 2016 07:27 AM, Andre LaBranche wrote:


On Jan 30, 2016, at 1:18 PM, Andre LaBranche > wrote:


This is not expected. Does memcached need to be configured for 
Unicode, maybe?


It looks like we expect to be in ascii mode, as far as memcached is 
concerned:


You can debug the memcache side of this by configuring calendarserver 
to not launch memcached; instead you'll do so manually, in the foreground.


Edit caldavd.plist, and insert the following Pools configuration under 
the Memcached dict:


  Pools
  
  Default
  
  MemcacheSocket
  
  ServerEnabled
  
  BindAddress
  127.0.0.1
  Port
  11211
  
  

Manually start memcached in verbose mode on 127.0.0.1:
memcached -l 127.0.0.1 -vv

Start calendar server, load a principal page: 
https://whatever:8443/principals/users/you


As you log in, you should see stuff in the memcached window:

<22 new auto-negotiating client connection
22: Client using the *ascii protocol*
<22 get DIGESTCREDENTIALS:...
>22 END


Also try the following test script which sets and gets unicode 
strings. bytes. whatever they are :) In SVN mode, I run ./bin/python 
mctest.py from the SVN dir to make sure the interpreter has access to 
six and memcache. If you don't use SVN mode, and don't have these 
modules installed system-wide, edit PYTHONPATH to help your 
interpreter find these modules.


#!/usr/bin/python
# -*- coding: latin-1 -*-
from __future__ import print_function

import six
from memcache import Client, SERVER_MAX_KEY_LENGTH, 
SERVER_MAX_VALUE_LENGTH


servers = ["127.0.0.1:11211"]
mc = Client(servers, debug=1)

def setget(key, val):
mc.set(key, val)
newval = mc.get(key)
print("key:", key)
print("set:".rjust(8), val)
print ("got:".rjust(8), newval, "\n")

setget("ascii", "xyzzy")

setget("unicode_1", six.u('\U0001f648'))
setget("unicode_2", six.u('\u25c9'))
setget("unicode_3", six.u('\u4f1a'))
setget("unicode_4", six.u('dr\\xe9'))

mc.disconnect_all()



Output looks like:

key: ascii
set: xyzzy
got: xyzzy

key: unicode_1
set: 
got: 

key: unicode_2
set: ◉
got: ◉

key: unicode_3
set: 会
got: 会

key: unicode_4
set: dré
got: dré


... and the memcached log confirms we're still in ascii mode:

<25 new auto-negotiating client connection
25: Client using the ascii protocol
<25 set ascii 0 0 5
>25 STORED
<25 get ascii
>25 sending key ascii
>25 END
<25 set unicode_1 0 0 4
>25 STORED
<25 get unicode_1
>25 sending key unicode_1
>25 END


-dre


Tried the experiment. Here is the output:

root@wheezy:/tmp# python test.py
key: ascii
set: xyzzy
got: xyzzy

key: unicode_1
set: 
got: 

key: unicode_2
set: ◉
got: ◉

key: unicode_3
set: 会
got: 会

key: unicode_4
set: dré
got: dré

and here is the memcached log:

<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 set ascii 0 0 5
>37 STORED
<37 get ascii
>37 sending key ascii
>37 END
<37 set unicode_1 0 0 4
>37 STORED
<37 get unicode_1
>37 sending key unicode_1
>37 END
<37 set unicode_2 0 0 3
>37 STORED
<37 get unicode_2
>37 sending key unicode_2
>37 END
<37 set unicode_3 0 0 3
>37 STORED
<37 get unicode_3
>37 sending key unicode_3
>37 END
<37 set unicode_4 0 0 4
>37 STORED
<37 get unicode_4
>37 sending key unicode_4
>37 END
<37 connection closed.


What next :)? I believe I could however safely go ahead and upload the 
new calendarserver version as this bug does not seem to breaking any 
functionality.


Thanks,
Rahul.


___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-30 Thread Andre LaBranche 

> On Jan 28, 2016, at 11:28 PM, Rahul Amaram 
>  wrote:
> ...
> (b) Finally, everything seems to be working as expected :). I am however, 
> seeing these two errors in my log files. Could I ignore them? Or is there a 
> fix for it?
> 
> 1. 2016-01-29 12:37:11+0530 [-] [caldav-1] [PooledMemCacheProtocol,client] 
> [twistedcaldav.memcachepool.MemCachePool#error] Memcache error: Invalid type 
> for key: , expecting a string; request: set 
> cacheToken:/calendars/__uids__/F343A113-0C2B-5A69-B049-A4C72FCAEF9B/ 
> 0553b3f8-1b1a-4ad9-a9d6-4aa70558d81e

This is not expected. Does memcached need to be configured for Unicode, maybe?

> 2. 2016-01-29 12:44:18+0530 [-] [twext.enterprise.jobs.queue#error] 
> workCheck: jobqueue is no longer overloaded

Not a problem as long as it's infrequent.

> (c) Also, could something be commented on the attached patch. Is is safe? I 
> really don't understand what the fix is but it does seem to help for handling 
> the problem of "/usr/bin/twistd: Unknown command: caldav_directoryproxy" 
> post-upgradation

The patch is baffling :) I don't know why you would need it... Maybe it is 
working around some difference in how you have installed our Python modules...

-dre
___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-30 Thread Rahul Amaram 


Thanks dre. This is resolved. Could you look at items (b) and (c) as 
well in my earlier mail?


Thanks,
Rahul.

On Saturday 30 January 2016 01:22 AM, Andre LaBranche wrote:

Hi,

psutil bumped to 3.4.2.

http://trac.calendarserver.org/changeset/15445/CalendarServer/trunk

All tests passed.

PASSED (skips=42, expectedFailures=32, unexpectedSuccesses=6, 
successes=2497)


-dre

On Jan 29, 2016, at 10:43 AM, Andre LaBranche > wrote:



On Jan 28, 2016, at 11:28 PM, Rahul Amaram 
> wrote:


(a) Could you check on this? It is sort of a blocker for pushing the 
new calendarserver version to Debian. If you could provide a patch, 
I could easily incorporate it.


Trunk has 3.2.2, which works and passes all our tests. I'm now trying 
running tests against psutil 3.4.1.


Looking through the change log, I'm guessing these are relevant:

http://trac.calendarserver.org/changeset/15281
http://trac.calendarserver.org/changeset/15282

-dre
___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org 


https://lists.macosforge.org/mailman/listinfo/calendarserver-dev




___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-30 Thread Andre LaBranche 

> On Jan 30, 2016, at 1:18 PM, Andre LaBranche  wrote:
> 
> This is not expected. Does memcached need to be configured for Unicode, maybe?

It looks like we expect to be in ascii mode, as far as memcached is concerned:

You can debug the memcache side of this by configuring calendarserver to not 
launch memcached; instead you'll do so manually, in the foreground.

Edit caldavd.plist, and insert the following Pools configuration under the 
Memcached dict:

  Pools
  
  Default
  
  MemcacheSocket
  
  ServerEnabled
  
  BindAddress
  127.0.0.1
  Port
  11211
  
  

Manually start memcached in verbose mode on 127.0.0.1:
memcached -l 127.0.0.1 -vv

Start calendar server, load a principal page: 
https://whatever:8443/principals/users/you 


As you log in, you should see stuff in the memcached window:

<22 new auto-negotiating client connection
22: Client using the ascii protocol
<22 get DIGESTCREDENTIALS:...
>22 END


Also try the following test script which sets and gets unicode strings. bytes. 
whatever they are :) In SVN mode, I run ./bin/python mctest.py from the SVN dir 
to make sure the interpreter has access to six and memcache. If you don't use 
SVN mode, and don't have these modules installed system-wide, edit PYTHONPATH 
to help your interpreter find these modules.

#!/usr/bin/python
# -*- coding: latin-1 -*-
from __future__ import print_function

import six
from memcache import Client, SERVER_MAX_KEY_LENGTH, SERVER_MAX_VALUE_LENGTH

servers = ["127.0.0.1:11211"]
mc = Client(servers, debug=1)

def setget(key, val):
mc.set(key, val)
newval = mc.get(key)
print("key:", key)
print("set:".rjust(8), val)
print ("got:".rjust(8), newval, "\n")

setget("ascii", "xyzzy")

setget("unicode_1", six.u('\U0001f648'))
setget("unicode_2", six.u('\u25c9'))
setget("unicode_3", six.u('\u4f1a'))
setget("unicode_4", six.u('dr\\xe9'))

mc.disconnect_all()



Output looks like:

key: ascii
set: xyzzy
got: xyzzy 

key: unicode_1
set: 
got:  

key: unicode_2
set: ◉
got: ◉ 

key: unicode_3
set: 会
got: 会 

key: unicode_4
set: dré
got: dré 


... and the memcached log confirms we're still in ascii mode:

<25 new auto-negotiating client connection
25: Client using the ascii protocol
<25 set ascii 0 0 5 
>25 STORED
<25 get ascii
>25 sending key ascii
>25 END
<25 set unicode_1 0 0 4 
>25 STORED
<25 get unicode_1
>25 sending key unicode_1
>25 END


-dre___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-29 Thread Andre LaBranche 

> On Jan 28, 2016, at 11:28 PM, Rahul Amaram 
>  wrote:
> 
> (a) Could you check on this? It is sort of a blocker for pushing the new 
> calendarserver version to Debian. If you could provide a patch, I could 
> easily incorporate it.

Trunk has 3.2.2, which works and passes all our tests. I'm now trying running 
tests against psutil 3.4.1.

Looking through the change log, I'm guessing these are relevant:

http://trac.calendarserver.org/changeset/15281 

http://trac.calendarserver.org/changeset/15282

-dre___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-29 Thread Andre LaBranche 
Hi,

psutil bumped to 3.4.2.

http://trac.calendarserver.org/changeset/15445/CalendarServer/trunk

All tests passed.

PASSED (skips=42, expectedFailures=32, unexpectedSuccesses=6, successes=2497)

-dre

> On Jan 29, 2016, at 10:43 AM, Andre LaBranche  wrote:
> 
> 
>> On Jan 28, 2016, at 11:28 PM, Rahul Amaram 
>> > > wrote:
>> 
>> (a) Could you check on this? It is sort of a blocker for pushing the new 
>> calendarserver version to Debian. If you could provide a patch, I could 
>> easily incorporate it.
> 
> Trunk has 3.2.2, which works and passes all our tests. I'm now trying running 
> tests against psutil 3.4.1.
> 
> Looking through the change log, I'm guessing these are relevant:
> 
> http://trac.calendarserver.org/changeset/15281 
> 
> http://trac.calendarserver.org/changeset/15282 
> 
> 
> -dre
> ___
> calendarserver-dev mailing list
> calendarserver-dev@lists.macosforge.org
> https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-28 Thread Rahul Amaram 



On Friday 29 January 2016 06:47 AM, Andre LaBranche wrote:

Also, the psutil version installed on the system is 3.4.1 while the version 
mentioned in requirements-stable.txt is 2.2.1. So, I went ahead and downgraded 
psutil to 2.2.1 by installing an older debian package and the error went away. 
So, psutil was the cause of the issue. Is it possible to make calendarserver 
compatible with 3.4.1? The incompatibility issue seems to be becoming a 
recurring theme :).

I can check on that, but I vaguely recall some reason why we haven't updated to 
newer psutil.
(a) Could you check on this? It is sort of a blocker for pushing the new 
calendarserver version to Debian. If you could provide a patch, I could 
easily incorporate it.

Finally, tasks were still not enabled for the user after the upgrade. What do I 
need to check for this?

I think this should be based only on the config value for EnableCardDAV, and 
possibly also the listener port configuration, depending on how the clients are 
configured.
I think tasks are related to caldavd. Anyway, I figured this out. The 
RestrictCalendarsToOneComponentType config has to be set to false for 
Thunderbird to be able to handle tasks and events. I set this option to 
false in the default config file that gets distributed with the package 
but for some reason while testing, I had removed the option. So, this 
problem could be considered resolved.


(b) Finally, everything seems to be working as expected :). I am 
however, seeing these two errors in my log files. Could I ignore them? 
Or is there a fix for it?


1. 2016-01-29 12:37:11+0530 [-] [caldav-1] 
[PooledMemCacheProtocol,client] 
[twistedcaldav.memcachepool.MemCachePool#error] Memcache error: Invalid 
type for key: , expecting a string; request: set 
cacheToken:/calendars/__uids__/F343A113-0C2B-5A69-B049-A4C72FCAEF9B/ 
0553b3f8-1b1a-4ad9-a9d6-4aa70558d81e
2016-01-29 12:37:11+0530 [-] [caldav-1] [PooledMemCacheProtocol,client] 
[twistedcaldav.memcachepool.MemCachePool#error] Memcache error: Invalid 
type for key: , expecting a string; request: set 
cacheToken:/calendars/__uids__/F343A113-0C2B-5A69-B049-A4C72FCAEF9B/calendar/ 
4582ad22-bb79-4c45-bdec-e7cae7aa5f80
2016-01-29 12:37:11+0530 [-] [caldav-1] [PooledMemCacheProtocol,client] 
[twistedcaldav.memcachepool.MemCachePool#error] Memcache error: Invalid 
type for key: , expecting a string; request: set 
cacheToken:/calendars/__uids__/F343A113-0C2B-5A69-B049-A4C72FCAEF9B/ 
40aed427-a35f-41dd-9f79-fefc56ff8006


2. 2016-01-29 12:44:18+0530 [-] [twext.enterprise.jobs.queue#error] 
workCheck: jobqueue is no longer overloaded


(c) Also, could something be commented on the attached patch. Is is 
safe? I really don't understand what the fix is but it does seem to help 
for handling the problem of "/usr/bin/twistd: Unknown command: 
caldav_directoryproxy" post-upgradation.



Looking forward to a response.

Thanks,
Rahul.
This fix is necessary for calendarserver_upgrade to work and probably for all of the other calendarserver_* utility scripts 
--- a/setup.py
+++ b/setup.py
@@ -329,7 +329,7 @@
 "pyOpenSSL>=0.14",# also for Twisted
 "service_identity",   # for Twisted
 "pycrypto",   # for Twisted
-"kerberos",
+"pykerberos",
 
 # Data store
 "xattr",
--- a/twisted/plugins/caldav.py
+++ b/twisted/plugins/caldav.py
@@ -50,5 +50,9 @@
 return self._serviceMaker.makeService(options)
 
 
-TwistedCalDAV = TAP("calendarserver.tap.caldav.CalDAVServiceMaker")
-DirectoryProxy = TAP("txdav.dps.server.DirectoryProxyServiceMaker")
+class T1(TAP):
+pass
+class T2(TAP):
+pass
+TwistedCalDAV = T1("calendarserver.tap.caldav.CalDAVServiceMaker")
+DirectoryProxy = T2("txdav.dps.server.DirectoryProxyServiceMaker")
___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-28 Thread Rahul Amaram 



On Thursday 28 January 2016 11:54 PM, Cyrus Daboo wrote:

Hi Andre,

--On January 28, 2016 at 10:21:50 AM -0800 Andre LaBranche 
 wrote:


It kind of looks like you're not running ./bin/develop after the 
upgrade,

before starting the service.


Right - you definitely need to do that because some of the Python 
packages have been updated - the 'NUM_CPUS' error results from an old 
version of the psutil package.


calendarserver is packaged. So, I cannot run ./bin/develop to install 
new packages. I have gone through the requirements-stable.txt and have 
ensured that all the libraries are at the requested minimum version. 
Does develop script does anything more which I should handle?


Also, the psutil version installed on the system is 3.4.1 while the 
version mentioned in requirements-stable.txt is 2.2.1. So, I went ahead 
and downgraded psutil to 2.2.1 by installing an older debian package and 
the error went away. So, psutil was the cause of the issue. Is it 
possible to make calendarserver compatible with 3.4.1? The 
incompatibility issue seems to be becoming a recurring theme :).


Finally, tasks were still not enabled for the user after the upgrade. 
What do I need to check for this?


Thanks,
Rahul.
___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-28 Thread Andre LaBranche 

> On Jan 28, 2016, at 11:57 AM, Rahul Amaram 
>  wrote:
> 
> 
> 
> On Thursday 28 January 2016 11:54 PM, Cyrus Daboo wrote:
>> Hi Andre,
>> 
>> --On January 28, 2016 at 10:21:50 AM -0800 Andre LaBranche  
>> wrote:
>> 
>>> It kind of looks like you're not running ./bin/develop after the upgrade,
>>> before starting the service.
>> 
>> Right - you definitely need to do that because some of the Python packages 
>> have been updated - the 'NUM_CPUS' error results from an old version of the 
>> psutil package.
>> 
> calendarserver is packaged. So, I cannot run ./bin/develop to install new 
> packages. I have gone through the requirements-stable.txt and have ensured 
> that all the libraries are at the requested minimum version. Does develop 
> script does anything more which I should handle?

For your use case, I don't think so.

> Also, the psutil version installed on the system is 3.4.1 while the version 
> mentioned in requirements-stable.txt is 2.2.1. So, I went ahead and 
> downgraded psutil to 2.2.1 by installing an older debian package and the 
> error went away. So, psutil was the cause of the issue. Is it possible to 
> make calendarserver compatible with 3.4.1? The incompatibility issue seems to 
> be becoming a recurring theme :).

I can check on that, but I vaguely recall some reason why we haven't updated to 
newer psutil.

> Finally, tasks were still not enabled for the user after the upgrade. What do 
> I need to check for this?

I think this should be based only on the config value for EnableCardDAV, and 
possibly also the listener port configuration, depending on how the clients are 
configured.

-dre
___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-28 Thread Andre LaBranche 
Hi,

What are your exact steps for the upgrade case?

It kind of looks like you're not running ./bin/develop after the upgrade, 
before starting the service.

-dre

> On Jan 28, 2016, at 10:16 AM, Rahul Amaram 
>  wrote:
> 
> Hi,
> 
> 1. If I upgrade the database from calendarserver 5.x to 7.x using 
> calendarserver_upgrade script, then I see the following error in error.log, 
> when trying to connect via caldav client (note that this does not happen for 
> fresh installation). Any help in debugging this would be appreciated.
> 
> ===
> 
> 2016-01-27 02:54:58+0530 [BinaryBoxProtocol,1,] Unhandled Error
>Traceback (most recent call last):
>  File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", 
> line 954, in _commandReceived
>deferred = self.dispatchCommand(box)
>  File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", 
> line 1011, in dispatchCommand
>return maybeDeferred(responder, box)
>  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", 
> line 150, in maybeDeferred
>result = f(*args, **kw)
>  File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", 
> line 1100, in doit
>return maybeDeferred(aCallable, **kw).addCallback(
>---  ---
>  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", 
> line 150, in maybeDeferred
>result = f(*args, **kw)
>  File "/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", 
> line 685, in logStats
>self.observer.logStats(stats)
>  File "/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", 
> line 378, in logStats
>self.systemStats = SystemMonitor()
>  File "/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", 
> line 587, in __init__
>"cpu count" : psutil.NUM_CPUS if psutil is not None else -1,
>exceptions.AttributeError: 'module' object has no attribute 'NUM_CPUS'
> 
> 2016-01-27 02:54:58+0530 [BinaryBoxProtocol,1,] Amp server or network failure 
> unhandled by client application.  Dropping connection!  To avoid, add 
> errbacks to ALL remote commands!
>Traceback (most recent call last):
>  File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", 
> line 978, in ampBoxReceived
>self._commandReceived(box)
>  File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", 
> line 957, in _commandReceived
>deferred.addCallback(self._safeEmit)
>  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", 
> line 317, in addCallback
>callbackKeywords=kw)
>  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", 
> line 306, in addCallbacks
>self._runCallbacks()
>---  ---
>  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", 
> line 588, in _runCallbacks
>current.result = callback(current.result, *args, **kw)
>  File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", 
> line 989, in _safeEmit
>aBox._sendTo(self.boxSender)
>  File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", 
> line 667, in _sendTo
>proto.transport.loseConnection()
>exceptions.AttributeError: 'DispatchingSender' object has no attribute 
> 'transport'
> 
> 2016-01-27 02:54:58+0530 [-] [caldav-1]  [BinaryBoxProtocol,client] 
> [calendarserver.accesslog#error] Unable to format event {'log_level': 
>

Re: [CalendarServer-dev] Error running calendarserver after upgrading db from 5.x to 7.x

2016-01-28 Thread Cyrus Daboo 

Hi Andre,

--On January 28, 2016 at 10:21:50 AM -0800 Andre LaBranche  
wrote:



It kind of looks like you're not running ./bin/develop after the upgrade,
before starting the service.


Right - you definitely need to do that because some of the Python packages 
have been updated - the 'NUM_CPUS' error results from an old version of the 
psutil package.


--
Cyrus Daboo

___
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev