Re: WebHelpers 0.6 released
in html.tags the form function has 'POST' for the default method, which is invalid xhtml. I changed it to: def form(url, method=post,... and all is well. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Bulk inserts
Depending on the locking behaviour of the bulk loader it could make sense to load the data into simple preprocessing tables which don't have any validation logic or primary key constraints. From these tables the final processing is then done. This would move the error handling from the bulk loader into the procedures inside the database, and solve bulk loading specific issues. We have also used this method to process ~10K record batch imports from a CSV file that is uploaded through a web interface. We were using MSSQL, but the theory would be the same and has worked very well for us. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Truncating an html string safely
On Jun 12, 5:13 am, Mike Orr [EMAIL PROTECTED] wrote: Although again, we have two issues. One is HTML-to-text (essentially lynx-as-a-function). The other is truncating an HTML string while keeping it well-formed (which means not stopping in the middle of a tag and closing any open tags). Here is another sanitizer (I think from something having to do with Zope): http://www.koders.com/python/fidFB51F4D2D89CC1397608213E09F11404D9B21059.aspx --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Truncating an html string safely
On Jun 12, 5:13 am, Mike Orr [EMAIL PROTECTED] wrote: Although again, we have two issues. One is HTML-to-text (essentially lynx-as-a-function). The other is truncating an HTML string while keeping it well-formed (which means not stopping in the middle of a tag and closing any open tags). Actually, I think we may have four issues...? 1) truncate HTML and end up with well-formed HTML. 2) strip all HTML tags (without an interest in text formatting) 3) html2text (trying to keep text formatting with p, block, etc.) 4) sanitizing HTML (not directly discussed here, but a good implementation of this will be helpful, increase security, and should be able to be extended trivially to provide #2, striping all HTML tags). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Bulk inserts
One other option might be to have each request dump its data to a random file with a .tmp extension to a queue directory. When the process is done, it can rename the file (which should be atomic) to a .csv file. Then the controller exits. You then have a second worker process checking the queue directory for .csv files and either a) process them into a larger holding file, which would then be bulk imported or b) just send them straight to the DB using the bulk insert command. You would just have to test to see which method was faster for you. Also, I am pretty sure MySQL's bulk import option allows you to specify more than one file to be imported, which might be a third option. Once the import is complete, the .csv files that have been imported get deleted. Since you have one worker process that isn't multi-threaded then a lot of the angst about file integrity should be alleviated. On Jun 11, 9:15 pm, Pete Wright [EMAIL PROTECTED] wrote: That's an awesome tip - thanks so much guys for pointing me in a far less painful direction. My only concern now is speed in working with the file and minimizing wait states. For example, I presumably don't want anything writing to the text file while I'm loading it into the database, and similarly I don't want to take time on every single request to check the size of the text file to see if I should run a LOAD DATA on it. So, I think what I'm going to end up doing here is converting time into seconds since midnight and then mod 60 it to come up with a file name suffix. I'll also have another process running separately processing and deleting unecessary files. Do you guys see any problems with that method? On Jun 11, 6:58 pm, Jonathan Vanasco [EMAIL PROTECTED] wrote: write to a .txt log in some sort of standardized format , when it hits 10k lines run a batch query as mentioned above, mysql has its own insert format; postgres does too. both insert formats are FAR faster than going through individual inserts. when i still used mysql about 4 years ago, i remember benching an insert of geographic data. it was something like 6 hours via inserts, 4hours via grouped inserts, and 25 seconds using the native import format. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Bulk inserts
if you use my method above I would do something like this: rename file to .err and send an email to an admin with as much detail as possible while possibly saving said error information in a separate file in case the email doesn't come through. The main concern here would be whether or not the records already inserted can be wrapped in a transaction and rolled back on error. I am guessing no. Which means you will need to manually figure out what the problem with the insert was, fix it, remove all the records that were already inserted, and then rename .err file to .csv for the worker thread to pick it up again. On Jun 11, 10:20 pm, Contact 42 [EMAIL PROTECTED] wrote: Jonathan Vanasco wrote: write to a .txt log in some sort of standardized format , when it hits 10k lines run a batch query what happens if an insert fails ? hd --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Cross-Project code sharing...modules/plugins what does Pylons use?
Hello all, I am new to the Python web-programming world and trying to decide on frameworks. I was really impressed with Django, but ran into some problems with IIS hosting. Since Pylons had really nice install instructions for IIS, I started to take a look and I like the philosophy. The WSGI from the ground up mentality seems to be a real plus. However, I have run across a show stopper for me with Pylons unless I have missed something in the documentation. Does Pylons support some kind of module/plugin architecture that will allow me to develop plug- in functionality across Pylons projects? What would be called in Django an app. For example, I would like to have a news, blog, and calendar module that I can plug into different applications. The goal is to have everything for the module contained in one subdirectory including any configuration, routing, templates, controllers, model, etc. So, something like this: /modules/news/... /modules/calendar/... /modules/blog/... Thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Cross-Project code sharing...modules/plugins what does Pylons use?
Thank you all for your comments. Its clear though, for me, that Pylons will not work for my projects. The need to develop components that fit easily and natively into the larger application is a must. Its disappointing though, b/c I was really liking the philosophy behind Pylons. Keep up the good work. IMO, this is a must for any robust framework and I hope Pylons incorporates this functionality into its framework philosophy some day. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Cross-Project code sharing...modules/plugins what does Pylons use?
On May 24, 9:13 pm, Philip Jenvey [EMAIL PROTECTED] wrote: If these are in fact separate applications, you can make each one its own Pylons app. Then you might also have a common package of code shared between all three (this would be a separate egg). Then at deployment time you can combine them into one composite application with Paste URL map, via the ini file: http://pythonpaste.org/deploy/#composite-applications Philip, Thanks for the reply. I guess I am not familiar enough with Pylons to know exactly what you mean by application. However, I don't think I intend for my modules to be applications in and of themselves. For example, I don't want to have to worry about defining DB connections, having decorator templates (i.e. the look that is shared across all pages), etc. per module. Rather, I would envision a module using the DB connection the application defined (even though it would have its own tables / database objects, etc.) and having generic template blocks that fit into the overall look of the site that is defined at the application level. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
