[Zope] Safari basic authentication problem
Hi Safari 4.0.2 fails to send an Authorization header to the server when the user is authenticated via basic authentication. This results in all sorts of permission problems. I realise this is not a Zope problem but perhaps I can hack some temporary solution server-side to convince Safari to play along. Header examples: Firefox: 'GET /sweet HTTP/1.1\r\n Host: 192.168.1.75:23190\r\n User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.1) Gecko/20090716 Ubuntu/9.04 (jaunty) Shiretoko/3.5.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n Accept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip,deflate\r\n Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n Keep-Alive: 300\r\n Connection: keep-alive\r\n Cookie: tree-s=eJzT0MgpMOQKVneEA1dbda4CI67EkgJjLj0AeGcHew\r\n Authorization: Basic YWRtaW46bG9jYWw=\r\n Cache-Control: max-age=0' Safari: --- 'GET /sweet HTTP/1.1\r\n Host: 192.168.1.75:23190\r\n User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1\r\n Referer: http://192.168.1.75:23190/sweet/pt_editForm\r\n Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\n Accept-Language: en-US\r\n Accept-Encoding: gzip, deflate\r\n Cache-Control: max-age=0\r\n Cookie: __utma=91023834.1699497027.1250064893.1250064893.1250064893.1; __utmb=91023834; __utmc=91023834; __utmz=91023834.1250064893.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); tree-s=eJzTyCkw5NLIKTDiClZ3hANXW3WuAmOuxEQ9AIOOB9Q\r\n Connection: keep-alive' This thread also discusses the issue: http://plope.com/Members/chrism/safari_3_discards_basic_auth It is fairly simple to replicate - add a Page Template to the root which displays request/AUTHENTICATED_USER and navigate to it (while logged in) with the respective browsers. Hedley ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] How to use ZPT with ad-hoc context?
Daniel Dekany wrote:
How to create a template context (here inside ZPT) that is not an
object from the ZODB, just a temporary object? This is what I tried:
class AdhocContext(Implicit):
pt = PageTemplateFile(whatever/path, globals())
...
MyZopeProduct:
def whatever(self):
Test
ctx = AdhocContext().__of__(self)
return ctx.pt()
The problem I have with this is that I can't access anything in
AdhocContext from the ZPT because the security manager blocks it (I
didn't forget to security.declarePublic + document what I wanted to
access). Is there a simple trick to solve this? (BTW, I will need to
invoke some Plone macros from that ZPT too... I hope that will just
work if this security matter is solved.)
You don't have to create the page template as an attribute of a class.
You can declare it as a local variable
pt = ZopeTwoPageTemplateFile('template.pt')
and then do
extra_context = {'context': some_context}
html = pt.pt_render(extra_context=extra_context)
The context variable in your template will then be what you want it to be.
If you still encounter security problems then add this line directly
after you declare class AdhocContext
__allow_access_to_unprotected_subobjects__ = 1
You probably should not use that line too often :)
Hedley
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Request time grows with memory size
On Mon, Apr 27, 2009 at 12:40 PM, Peter Bengtsson pete...@gmail.com wrote: What have you done to investigate memory leaks? What external connectors are you using, like MySQL or LDAP? It is probably not a memory leak. The graph is what I'd expect in a garbage collection scenario (ie. Python). Hedley ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Building a fast, scalable yet small Zope application
I've followed this thread with interest since I have a Zope site with tens of millions of entries in BTrees. It scales well, but it requires many tricks to make it work. Roche Compaan wrote these great pieces on ZODB, Data.fs size and scalability at http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/catalog-indexes and http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/fat-doesnt-matter . My own in-house product is similar to GoogleAnalytics. I have to use a cascading BTree structure (a btree of btrees of btrees) to handle the volume. This is because BTrees do slow down the more items they contain. This is not a ZODB limitation or flaw - it is just how they work. My structure allows for fast inserts, but they also allow aggregation of data. So if my lowest level of BTrees store hits for a particular hour in time then the containing BTree always knows exactly how many hits were made in a day. I update all parent BTrees as soon as an item is inserted. The cost of this operation is O(1) for every parent. These are all details but every single one influenced my design. What is important is that you cannot just use the ZCatalog to index tens of millions of items since every index is a single BTree and will thus suffer the larger it gets. So you must roll your own to fit your problem domain. Data warehousing is probably a good idea as well. My problem domain allows me to defer inserts, so I have a queuerunner that commits larger transactions in batches. This is better than lots of small writes. This may of course not fit your model. Familiarize yourself with TreeSets and set operations in Python (union etc.) since those tools form the backbone of catalogueing. Hedley ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope-dev] zopectl does not terminate
My apologies. I sent this to the wrong list. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope] zopectl does not terminate
Hi all I run my script foo.zctl with zopectl run foo.ctl param1 param2. This script operates on a large ZODB and catches ConflictErrors accordingly. It iterates over a set, updates data and commits the transaction every 100 iterations. But I've noticed two things: 1. ConflictErrors are never fully caught. The show up in the console (this is acceptable I suppose), but my script stops executing on the conflict and does not continue. The zope process stays alive. 2. In the event of no conflict errors my script executes its last line (print 'done') but the process does not always terminate. If I instruct my script to not update the ZODB at all it terminates without problems. I'm running it on a live site with 7 ZEO clients. I've stopped a client (say client 2) so it is not accessed concurrently and run my script with client2/zopectl. It is in fact a Plone site but that should be irrelevant. Thanks for any help Hedley ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] zopectl does not terminate
Hi Tres! Thanks for the tips. I managed to get my script running in batches and with manual intervention. When in future I encounter the same problems I'll report back to this thread. Hedley ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope-dev] zopectl does not terminate
Hi all I run my script foo.zctl with zopectl run foo.ctl param1 param2. This script operates on a large ZODB and catches ConflictErrors accordingly. It iterates over a set, updates data and commits the transaction every 100 iterations. But I've noticed two things: 1. ConflictErrors are never fully caught. The show up in the console (this is acceptable I suppose), but my script stops executing on the conflict and does not continue. The zope process stays alive. 2. In the event of no conflict errors my script executes its last line (print 'done') but the process does not always terminate. If I instruct my script to not update the ZODB at all it terminates without problems. I'm running it on a live site with 7 ZEO clients. I've stopped a client (say client 2) so it is not accessed concurrently and run my script with client2/zopectl. It is in fact a Plone site but that should be irrelevant. Thanks for any help Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Next Step to Bug Resolution???
Hi Tim I'm more involved with Plone but can provide a slightly more digestible answer :) Chris mentioned unit tests. You do not have to write a new unit test. What is required is to have a look at the tests and then identify one that is relevant to your problematic method. This test should be very easy to read. The hard part is for you to expand this test to demonstrate the exception. This will typically require a few lines of code but since you already fixed a bug I assume it won't be hard to do. Then you have to run all the other tests to check that you haven't broken anything else by doing ./bin/instance test -s zope.something. Finally create a patch file and attach it to your original report on the tracker. Mmm, after writing all this I can see why it is a pain. I do however understand the need for unit tests so I don't think you are going to be able to get past that bit. Perhaps a kind Zope dev will pick up on this. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Run a few tests and let me know please. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. And the cache is persistent as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. To answer the question: I believe catalogcache will win every time since the return time of a cached query is not dependent on the complexity of the query. We should get a few benchmarks running at query level. I'll have a bit of time next week. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Method search just inspects queries and stores results to memcache. Can you give me an example where the cache would deliver non-committed data? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, Oct 25, 2008 at 2:57 PM, Andreas Jung [EMAIL PROTECTED] wrote: On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas Thanks Andreas. I'll have a look at your code when available. Christian, I do have a mistake in my reasoning. If an object is added to the catalog in a transaction and I cache that object as result of a query in that same transaction, and then the transaction fails I'll have a bad cache. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Kind of like how I came to the same conclusion in parallel to you and stuffed up this thread :) Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. So this is the area I have to focus on right now. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. A possible solution is to keep a module level cache which can be committed to the memcache on transaction boundaries. That way I'll incur no performance penalty. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog caching with memcached
Hi all The past few weeks I've been optimizing a busy Plone site and so collective.catalogcache was born. It uses memcached as a distributed ZCatalog cache. It is currently in production and seems to be holding just fine. The site went from being unusable to serving quite a bit of data. I'll modify it in future to not be too tied to just memcached, but that's still some way off. BTW, if you do not have memcached available the catalog functions as usual. It's only been tested for Zope 2.9.6 - 2.10.6. The repo is at http://dev.plone.org/collective/browser/collective.catalogcache. The URL may contain the word Plone but it's all Zope :) The readme contains more info and instructions. The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
