[web2py] Re: Should we have a feature freeze and stability maintenance period in future?
I'm thinking a few possibilities... we can, as a possible option, give folks the opportunity to either register there install of web2py with additional user registration option (depending of your philosophy wrt that) - which could also serve as tracking mechanism to encourage people to keep up with updates (i.e. Flash screen: you are on version 1.62. Latest stable release is 1.84. Would you like to update now? y/n/). Or maybe offer the option to register as a web2py user only (which may help augment numbers). Perhaps users can be motivated to register? For example register and get access to the web2py community, ask questions, get real case answers, LOG bugs, etc. which would be a hard requirement for this type of activity) registration: * install/user registration on install (I.e. .msi on windows) * on launch of app admin * on error? (I.e. Exception is thrown, user can be prompted to send error to web2py.com db.errors to 1) report the problem b) if issue can be figured out automatically like one I seem to be getting lately where AMF fails on import (missing class) when I could actually see it ;) - just as an example :)) then 2) the system can return resolution if available, or automatically log a ticket (oh, and while were there, why not offer the option of registration?) * obviously on launch of web2py bug tracker * on web2py software update * maybe a nice feature when installing one of those cool fee apps, user gets prompted with option to register * etc... I think there are many ways to get user attention to register with valid Google account for reducing the risk of spam with bug logging (and can have a bit of value add as well - again if inline with your web2py philosophy). web2py already has great authentication capabilities with variety and ease, leveraging that should be do-able? Also, i assume it should also be possible to query Google DB for user authentication, if not user validation is done through web2py.com? Mart :) On Aug 26, 7:23 pm, mdipierro mdipie...@cs.depaul.edu wrote: This is reasonable but how to check? On Aug 26, 6:18 pm, mart msenecal...@gmail.com wrote: Interesting point... How about bugs can only be truly submitted if user is registered in web2py user group? Can we up authentication detail for users (if required)... True that it could be scary that folks start logging bugs left and right... If part of user group, would this only generate same amount of traffic? Mart :) On Aug 26, 4:56 pm, Michele Comitini michele.comit...@gmail.com wrote: some thoughts... Firstly it is important that tickets reach their fate (closed, wont fix, cant fix, not a bug, whatever ) soon, otherwise it would be worst than people writing for bugs already fixed. You can see many projects that have 3 year old tickets... This means that some individuals should handle the task of keeping an eye on tickets that stall and take actions. Secondly I would suggest that we put down a little howto on reporting bugs. Short, no one reads it otherwise. ;-) The answer to a user signalling something that sounds as a bug could be, please read web2py.com/bugshowto and report the problem, thanks!. I like the remote thing, but should it require a captha or similar, to avoid spammers and to make clear to the sender that something is going out of its machine? Else it would look much like those desktop applications that ask you to send info to someone you do not know, not only, you do not even know what data is going to be sent! Just YES or NO! Scaring stuff! ;-) Those ticket could contain really sensitive data! Has anyone already managed a similar situation? please speakup! mic 2010/8/26 mart msenecal...@gmail.com: good plan! Do you have people dedicated to test purposes for the monitoring help? Were you thinking on a something like a 2 tier filtering system? first level bug validation (is this really a new bug?) as automated filtering second: trusted testers/dev folks who can best help in filtering and pass best info to you? open a web2py ticket from the admin! Very innovative! Lots of possibilities there! :). Also, do let me know if you would like to look at (or have thoughts on) branching strategies as a means to allow testing phases while continuing with on going development. Thanks, Mart :) On Aug 26, 8:57 am, mdipierro mdipie...@cs.depaul.edu wrote: I am in favor of both proposals 1) user google code more 2) having a bug tracking application but - I would like some help in monitoring this list and opening bugs on google code when a new bug is suggested on the list. This is because I do not want to have to explain to new users how to open google code tickets and I do not want to have to open them myself (i.e. open,fix,close vs just fix it). - If we are doing this we should give it an edge: why not add to the web2py
[web2py] Re: fascinating
Each time the site engine was crashing (not web2py, long ago) the stack trace was emailed to me along with all variables values that existed at all levels of the trace. good method to improve web apps development , getting feedback helping to minimize bugs interesting...
[web2py] Re: fascinating
REF: issue/bug loging:
useful for auto-send-error-to-server for error collecting ? (issue
tracking for validation on IS_BUG). Anyways, looks exciting, think
I'll play with this module for a while... Thanks for sharing!
Mart :)
On Aug 26, 11:26 pm, Alexey Nezhdanov snak...@gmail.com wrote:
Yes, once I used it to print an 'enhanced stack backtrace'. Each time the
site engine was crashing (not web2py, long ago) the stack trace was emailed
to me along with all variables values that existed at all levels of the
trace.
On Fri, Aug 27, 2010 at 6:53 AM, GoldenTiger goldenboy...@gmail.com wrote:
do you refer to module inspect? it's great for introspecting on live
objects and their source code
On 27 ago, 03:44, mdipierro mdipie...@cs.depaul.edu wrote:
I discovered this:
def f(name):
import inspect
c_frame = inspect.getouterframes(inspect.currentframe(), 1)[1][0]
c_args, c_varargs, c_varkw, c_locals =
inspect.getargvalues(c_frame)
d = dict(c_locals)
return d[name]
def g():
a=5
print f('a')
g() # prints 5
[web2py] Re: fascinating
I think the inspect module could be useful in deployed production
code: if you have a function with side effects, such as:
def update_and_return(value):
# 1. store value in database
# 2. read another value from database
# 3. do some very heavy calculations based on the passed
#value and the one you just read in from the database
...
return calculated_result
Of course this is a trivial example (and you would simply split it
into two functions), though using inspect, it'd be possible to
determine if the return value is *ever* used. This includes being
assigned to a variable that's ever read (this check would be disabled
during interactive use), or if it's passed into another function, ala
foo(update_and_return(5)), that ever uses that variable (with
special considerations for *args, **kwargs, and other constructs).
Such functionality would be nice to have functools as an optimized c-
routine, and could be used as such:
from functools import returning # this doesn't actually exist
def update_and_return(value):
# 1. store value in database
if not returning():
return
# 2. read another value from database
# 3. do some very heavy calculations based on the passed
#value and the one you just read in from the database
...
return calculated_result
The ast module is also worth looking into. I was trying to use it to
leverage python's already pseudo-code like qualities for user
customization in an grad-level CS theory class assignment by allowing
(the grader) to code a very abstract problem definition (without even
the optimizations you do subconsciously). The program would leverage
the interpreter to do the parsing, and then I would rearrange and
prune the parse tree in various ways, most commonly by inspecting the
code of boolean filter functions (which of the elements are worth
considering?) that took potentially millions of input elements and
returned a few dozen outputs, and creating generators that only
yielded those same few dozen elements without having to scan the input
sequence (based on rules governing the input elements, and assumptions
about the supplied filter functions).
Let's just say it turned out to be a lot faster than programs from
those students that were counting on C++ being fast enough for such
brute-force code -- pity that much of academia these days consists of
students that only know (or even know of) one programming language
(some of which believe that, like the Windows and Mac of the
computing world, there are only a handful of choices out there).
On Aug 26, 8:53 pm, GoldenTiger goldenboy...@gmail.com wrote:
do you refer to module inspect? it's great for introspecting on live
objects and their source code
On 27 ago, 03:44, mdipierro mdipie...@cs.depaul.edu wrote:
I discovered this:
def f(name):
import inspect
c_frame = inspect.getouterframes(inspect.currentframe(), 1)[1][0]
c_args, c_varargs, c_varkw, c_locals =
inspect.getargvalues(c_frame)
d = dict(c_locals)
return d[name]
def g():
a=5
print f('a')
g() # prints 5
[web2py] Problem with Oracle connection string...
Hello,
I have a problem when I try to connect to an Oracle database with this
connection string:
db = DAL(oracle://username:password@//network_server:1521/
database_name)
Traceback (most recent call last):
File C:\MyApps\web2py\gluon\restricted.py, line 186, in restricted
exec ccode in environment
File C:/MyApps/web2py/applications/test1/models/db.py, line 16, in
module
db = DAL(oracle://username:password@//network_server:1521/
database_name)
File C:\MyApps\web2py\gluon\sql.py, line 3992, in DAL
raise RuntimeError, %s (tried 5 times) % exception
RuntimeError: argument 1 must be unicode, not str (tried 5 times)
3987. except SyntaxError, exception:
raise SyntaxError, exception
except Exception, exception:
if uri==uris[-1]:
time.sleep(1)
raise RuntimeError, %s (tried 5 times) % exception
if __name__ == '__main__':
import doctest
doctest.testmod()
* exception: TypeError('argument 1 must be unicode, not str',)
* builtinRuntimeError: type 'exceptions.RuntimeError'
Thank you.
i
[web2py] Re: Facebook oauth
I think so but I did not try it.
On Aug 26, 11:40 pm, Narendran gunanar...@gmail.com wrote:
Hello Michele/Massimo,
Has this fix been taken in? Is it available in any web2py build now?
On Aug 22, 4:10 am, Michele Comitini michele.comit...@gmail.com
wrote:
Massimo here is the full patch with modifications for
oauth*_account.py login_methods
tnx!
2010/8/22 Michele Comitini michele.comit...@gmail.com:
in tools.py line 1436:
next = URL(r=request) + '?_next=' + next
should be like this?
next = self.url('user', args='login') + '?_next=' + next
2010/8/22 Michele Comitini michele.comit...@gmail.com:
Ok found a way to fix the problem with redirection (_next parameter).
Massimo pls look if the following is correct!
diff -r 9261ce4eda7f gluon/tools.py
--- a/gluon/tools.py Thu Aug 19 04:13:54 2010 +0200
+++ b/gluon/tools.py Sun Aug 22 00:08:25 2010 +0200
@@ -982,7 +983,7 @@
request = self.environment.request
args = request.args
if not args:
- redirect(self.url(args='login'))
+ redirect(self.url(args='login', vars=request.vars))
elif args[0] in self.settings.actions_disabled:
raise HTTP(404)
if args[0] == 'login':
2010/8/20 Michele Comitini michele.comit...@gmail.com:
Hello Narendran,
Do not use that it is old..
now facebook is supported inside web2py distribution with
oauth20_account.py
you can find an example app here:
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
for a simple example usage of graph api look here:
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
for the redirection after login I am investigating...
mic
2010/8/20 Narendran gunanar...@gmail.com:
Hello all,
I picked the Facebook oauth submitted by mcm from
https://code.google.com/r/michelecomitini-facebookaccess/source/brows...
(referred in this thread:
http://groups.google.com/group/web2py/browse_thread/thread/be441047bf...)
I am facing one isse:
1. I've placed require_login decoration on a method say a(). After the
authentication is complete, the page always gets redirected to default/
index, whereas the ideal behaviour would be to go to a(). It works as
intended if I use default auth instead of Facebook auth.
Also, is there any plugin/package that would provide full-fledged
support for using Facebook Graph API with web2py?
--
Thanks
Narendran
oauth_action_flow_fix.diff
17KViewDownload
[web2py] Re: Problem with Oracle connection string...
That is not the sytnax for oracle uris. The oracle syntax is:
DAL(''oracle://username:passw...@database' )
The oracle driver does not support connecting to a remote server. You
must be run a oracle client locally.
On Aug 27, 8:06 am, ionel ionelanton...@gmail.com wrote:
Hello,
I have a problem when I try to connect to an Oracle database with this
connection string:
db = DAL(oracle://username:password@//network_server:1521/
database_name)
Traceback (most recent call last):
File C:\MyApps\web2py\gluon\restricted.py, line 186, in restricted
exec ccode in environment
File C:/MyApps/web2py/applications/test1/models/db.py, line 16, in
module
db = DAL(oracle://username:password@//network_server:1521/
database_name)
File C:\MyApps\web2py\gluon\sql.py, line 3992, in DAL
raise RuntimeError, %s (tried 5 times) % exception
RuntimeError: argument 1 must be unicode, not str (tried 5 times)
3987. except SyntaxError, exception:
raise SyntaxError, exception
except Exception, exception:
if uri==uris[-1]:
time.sleep(1)
raise RuntimeError, %s (tried 5 times) % exception
if __name__ == '__main__':
import doctest
doctest.testmod()
* exception: TypeError('argument 1 must be unicode, not str',)
* builtinRuntimeError: type 'exceptions.RuntimeError'
Thank you.
i
[web2py] Table field: compute with 'id' ?
Hello,
I am trying to generate a field with a default value based on the
record id -- but it doesn't work. For example:
db.define_table('article',
Field('designation', type='string', length=200, required=True,
requires=IS_NOT_EMPTY()),
Field('details', type='string', length=2000),
Field('intcodebarre', type='string', length=50, compute=lambda r:
SolA%-.6d % r['id']),
format='%(designation)s')
Is not accepted (but works for a declared field). For a future patch
or bad design on my side?
BR, Andre
[web2py] Re: Table field: compute with 'id' ?
it cannot be done because the computation is done before the insert
while the id is assigned by the database after the insert.
On Aug 27, 8:32 am, dederocks dediro...@gmail.com wrote:
Hello,
I am trying to generate a field with a default value based on the
record id -- but it doesn't work. For example:
db.define_table('article',
Field('designation', type='string', length=200, required=True,
requires=IS_NOT_EMPTY()),
Field('details', type='string', length=2000),
Field('intcodebarre', type='string', length=50, compute=lambda r:
SolA%-.6d % r['id']),
format='%(designation)s')
Is not accepted (but works for a declared field). For a future patch
or bad design on my side?
BR, Andre
[web2py] Re: doc2py - php.net like python module documentation
Hello, have a look at the Sphinx web app from this years GSoC: http://gsoc.jacobmason.us/demo/contents Done by jacob Mason (http://gsoc.jacobmason.us/blog/?cat=3). This is a good opportunity to make web2py docs commenatble online.
[web2py] Re: doc2py - php.net like python module documentation
Actually, the plan was to make the docs editable online but move from reST to MARKMIN and use our own software. On Aug 27, 9:38 am, Timmie timmichel...@gmx-topmail.de wrote: Hello, have a look at the Sphinx web app from this years GSoC: http://gsoc.jacobmason.us/demo/contents Done by jacob Mason (http://gsoc.jacobmason.us/blog/?cat=3). This is a good opportunity to make web2py docs commenatble online.
[web2py] handing a file upload submitted outside of web2py
I'm using Pyjamas to build my client app and Web2py to build the server. I'm submitting a HTML form (generated by Pyjamas) to the Web2py server code. Where should I look to see how I can handle the submission using Web2py? I already have a function defined in /app/controllers/default.py which gets called with the form data from the web browser (stuff like request.vars). The return value from my function is correctly returned to the client but obviously I need to access the file the user selected and store it in the database (I'm using GAE so storing in the database is the route I need to take) Any pointers?
Re: [web2py] Re: Web2py and threads
On Aug 25, 2010, at 8:12 PM, Jonathan Lundell wrote: On Aug 25, 2010, at 7:56 PM, mdipierro wrote: This is a bug. I fixed it in trunk. Thanks Jonathan. It's fixed in the sense that it won't raise an exception. But now how is calling _unlock different from calling forget? Nag. On Aug 25, 9:30 pm, Jonathan Lundell jlund...@pobox.com wrote: On Aug 25, 2010, at 6:37 PM, mdipierro wrote: The problem is only if have two http request from the same client in the same session Thanks for that; I was wondering under which conditions unlocking might be permissible (and I'm still not entirely clear, but never mind for now). My concern is this. Here's unlock: def _unlock(self, response): if response and response.session_file: try: portalocker.unlock(response.session_file) response.session_file.close() del response.session_file - except: ### this should never happen but happens in Windows pass Now we save the session file: def _try_store_on_disk(self, request, response): if response._dbtable_and_field \ or not response.session_id \ or self._forget: self._unlock(response) return if response.session_new: # Tests if the session folder exists, if not, create it session_folder = os.path.dirname(response.session_filename) response.session_file = open(response.session_filename, 'wb') portalocker.lock(response.session_file, portalocker.LOCK_EX) cPickle.dump(dict(self), response.session_file) self._unlock(response) But response.session_file is None at this point. A arrives loads session and unlocks B arrives loads session and unlocks A change session and saves it B changes session and saves it Nothing breaks but B never sees changes made by A and they are overwritten by B. With locks A arrives loads session B arrives and waits A change session and saves it B loads session (with changes made by A) B changes session and saves it On Aug 25, 3:52 pm, Jonathan Lundell jlund...@pobox.com wrote: On Aug 25, 2010, at 1:41 PM, mdipierro wrote: call session._unlock() if you do not need session locking If you do that (without calling session.forget), what will happen in _try_store_on_disk when cPickle.dump(dict(self), response.session_file) is called with a None file argument? Or is cPickle.dump cool with that? Or am I misreading the logic? On Aug 25, 11:38 am, Phyo Arkar phyo.arkarl...@gmail.com wrote: Yes may be session was locked , thats why session.current=processing_path not working But then again , while processing files i try opening separate page , to other controller , it was waited till the first (file Crawler) page finished parsing. ok i will make a separate thread about this. On 8/25/10, mdipierro mdipie...@cs.depaul.edu wrote: On Aug 25, 11:00 am, Phyo Arkar phyo.arkarl...@gmail.com wrote: Did I Read that reading files inside controller will block web2py , Does it? No web2py does not block. web2py only locks sessions that means one user cannot request two concurrent pages because there would be a race condition in saving sessions. Two user can request different pages which open the same file unless the file is explicitly locked by your code. Thats a bad news.. i am doing a file crawler and while crawling , web2py is blocked even tho the process talke only 25% of 1 out of 4 CPUs .. Tell us more or I cannot help. On 8/25/10, pierreth pierre.thibau...@gmail.com wrote: I would appreciate a good reference to understand the concepts you are talking about. It is something new to me and I don't understand. On 25 août, 11:22, John Heenan johnmhee...@gmail.com wrote: No, nothing that abstract. Using WSGI forces a new thread for each request. This is is a simple and inefficient brute force approach that really only suits the simplest Python applications and where only a small number of concurrent connection might be expected. Any application that provides web services is going to OS block on file reading (and writing) and on database access. Using threads is a classic and easy way out that carries a lot of baggage. Windows has had a way out of this for years with its asynch (or event) notification set up through an OVERLAPPED structure. Lightttpd makes use of efficient event notification schemes like kqueue and epoll. Apache only uses such schemes for listening and Keep- Alives. No matter how careful one is with threads and processes there always appears to be unexpected gotchas. Python has a notorious example, the now fixed 'Beazly Effect' that affected the GIL. Also I don't think there is a single experienced Python user that trusts the GIL. John Heenan
[web2py] Re: handing a file upload submitted outside of web2py
do you want to access to the image uploaded with form for example?.
use download function in controller
img src={{=URL('download', args=item.image)}}
width=200px /
On Aug 27, 12:08 pm, Carl carl.ro...@gmail.com wrote:
I'm using Pyjamas to build my client app and Web2py to build the
server.
I'm submitting a HTML form (generated by Pyjamas) to the Web2py server
code.
Where should I look to see how I can handle the submission using
Web2py?
I already have a function defined in /app/controllers/default.py which
gets called with the form data from the web browser (stuff like
request.vars).
The return value from my function is correctly returned to the client
but obviously I need to access the file the user selected and store it
in the database (I'm using GAE so storing in the database is the route
I need to take)
Any pointers?
[web2py] Re: Web2py and threads
You are right. Please check trunk again. Massimo On Aug 27, 10:25 am, Jonathan Lundell jlund...@pobox.com wrote: On Aug 25, 2010, at 8:12 PM, Jonathan Lundell wrote: On Aug 25, 2010, at 7:56 PM, mdipierro wrote: This is a bug. I fixed it in trunk. Thanks Jonathan. It's fixed in the sense that it won't raise an exception. But now how is calling _unlock different from calling forget? Nag. On Aug 25, 9:30 pm, Jonathan Lundell jlund...@pobox.com wrote: On Aug 25, 2010, at 6:37 PM, mdipierro wrote: The problem is only if have two http request from the same client in the same session Thanks for that; I was wondering under which conditions unlocking might be permissible (and I'm still not entirely clear, but never mind for now). My concern is this. Here's unlock: def _unlock(self, response): if response and response.session_file: try: portalocker.unlock(response.session_file) response.session_file.close() del response.session_file - except: ### this should never happen but happens in Windows pass Now we save the session file: def _try_store_on_disk(self, request, response): if response._dbtable_and_field \ or not response.session_id \ or self._forget: self._unlock(response) return if response.session_new: # Tests if the session folder exists, if not, create it session_folder = os.path.dirname(response.session_filename) response.session_file = open(response.session_filename, 'wb') portalocker.lock(response.session_file, portalocker.LOCK_EX) cPickle.dump(dict(self), response.session_file) self._unlock(response) But response.session_file is None at this point. A arrives loads session and unlocks B arrives loads session and unlocks A change session and saves it B changes session and saves it Nothing breaks but B never sees changes made by A and they are overwritten by B. With locks A arrives loads session B arrives and waits A change session and saves it B loads session (with changes made by A) B changes session and saves it On Aug 25, 3:52 pm, Jonathan Lundell jlund...@pobox.com wrote: On Aug 25, 2010, at 1:41 PM, mdipierro wrote: call session._unlock() if you do not need session locking If you do that (without calling session.forget), what will happen in _try_store_on_disk when cPickle.dump(dict(self), response.session_file) is called with a None file argument? Or is cPickle.dump cool with that? Or am I misreading the logic? On Aug 25, 11:38 am, Phyo Arkar phyo.arkarl...@gmail.com wrote: Yes may be session was locked , thats why session.current=processing_path not working But then again , while processing files i try opening separate page , to other controller , it was waited till the first (file Crawler) page finished parsing. ok i will make a separate thread about this. On 8/25/10, mdipierro mdipie...@cs.depaul.edu wrote: On Aug 25, 11:00 am, Phyo Arkar phyo.arkarl...@gmail.com wrote: Did I Read that reading files inside controller will block web2py , Does it? No web2py does not block. web2py only locks sessions that means one user cannot request two concurrent pages because there would be a race condition in saving sessions. Two user can request different pages which open the same file unless the file is explicitly locked by your code. Thats a bad news.. i am doing a file crawler and while crawling , web2py is blocked even tho the process talke only 25% of 1 out of 4 CPUs .. Tell us more or I cannot help. On 8/25/10, pierreth pierre.thibau...@gmail.com wrote: I would appreciate a good reference to understand the concepts you are talking about. It is something new to me and I don't understand. On 25 août, 11:22, John Heenan johnmhee...@gmail.com wrote: No, nothing that abstract. Using WSGI forces a new thread for each request. This is is a simple and inefficient brute force approach that really only suits the simplest Python applications and where only a small number of concurrent connection might be expected. Any application that provides web services is going to OS block on file reading (and writing) and on database access. Using threads is a classic and easy way out that carries a lot of baggage. Windows has had a way out of this for years with its asynch (or event) notification set up through an OVERLAPPED structure. Lightttpd makes use of efficient event notification schemes like kqueue and epoll. Apache only uses such schemes for listening and Keep- Alives. No matter how careful one is with threads and processes there always appears to be unexpected gotchas. Python has a notorious example,
[web2py] Re: list:string not working
Massimo, that gave me a ticket with this:
Traceback (most recent call last):
File C:\repo\anima\gluon\restricted.py, line 186, in restricted
exec ccode in environment
File C:/repo/anima/applications/welcome/controllers/default.py,
line 754, in module
File C:\repo\anima\gluon\globals.py, line 96, in lambda
self._caller = lambda f: f()
File C:/repo/anima/applications/welcome/controllers/default.py,
line 553, in editar_diag_beta
form =
crud.update(db.diagnostico_beta,request.args(0),deletable=False)
File C:\repo\anima\gluon\tools.py, line 2756, in update
hideerror=self.settings.hideerror):
File C:\repo\anima\gluon\sqlhtml.py, line 906, in accepts
hideerror=hideerror,
File C:\repo\anima\gluon\html.py, line 1512, in accepts
status = self._traverse(status,hideerror)
File C:\repo\anima\gluon\html.py, line 522, in _traverse
newstatus = c._traverse(status,hideerror) and newstatus
File C:\repo\anima\gluon\html.py, line 522, in _traverse
newstatus = c._traverse(status,hideerror) and newstatus
File C:\repo\anima\gluon\html.py, line 522, in _traverse
newstatus = c._traverse(status,hideerror) and newstatus
File C:\repo\anima\gluon\html.py, line 522, in _traverse
newstatus = c._traverse(status,hideerror) and newstatus
File C:\repo\anima\gluon\html.py, line 529, in _traverse
newstatus = self._validate()
File C:\repo\anima\gluon\html.py, line 1300, in _validate
(value, errors) = validator(value)
ValueError: too many values to unpack
But I had already succeed with this:
class TAGS_LIST:
def __init__(self, separator=',', error_message='This is not a
valid list!'):
self.separator = separator
self.e = error_message
def __call__(self,value):
try:
list = value.split(self.separator)
return (list, None)
except:
return (value, self.e)
def formatter(self, value):
tags = ''
for tag in value:
tags += '%(tag)s%(sep)s ' %
{'tag':tag,'sep':self.separator}
return tags
I have two question:
Could this be general relating different DBs?
Is there a way to define a default widget for a custom validator?
On Aug 26, 7:28 pm, mdipierro mdipie...@cs.depaul.edu wrote:
try define
class IS_LIST_OF_STRINGS:
def __call__(self,value):
return [str(x) for x in value.split(',')]
def formatter(self,value):
return ', '.join(value)
and use
Field(...,'list:string',requires=IS_LIST_OF_STRINGS())
On Aug 26, 4:27 pm, yamandu yamandu.co...@gmail.com wrote:
Yes, I wish I could let the user input strings that don´t belong to
predetermined set.
A type of list that can add new itens does not make much sense for me.
This is just a multi select, not properly a list in its most
meaningful sense.
On Aug 26, 5:36 pm, mdipierro mdipie...@cs.depaul.edu wrote:
This is not wrong.
The problem is that your field has type='list:string' but you did not
set requires=IS_IN_SET(('aaa','bbb','ccc')) or requires=IS_IN_DB(...)
Without the validator web2py does not know which options are valid and
which ones are not and does not know how to make a dropbox.
You have a valid point though. There should be a default validator
that allows to write strings separated by a comma or something like
plugin tagging. Such validator has not yet been created.
On Aug 26, 3:27 pm, yamandu yamandu.co...@gmail.com wrote: I possibly
found a problem with list:string field.
I´ve searched for a widget and I could found one for it.
So I starte to make my own based on
thishttp://blog.crazybeavers.se/wp-content/demos/jquery.tag.editor/
I could be like that tag editor in plugin_wiki too but I found it
harder to base on it.
The problem is when you update a field of type list:string using the
item1|item2|... syntax it parses correctly and saves like array
['item1','item2']
But if you update the record it shows like ['item1','item2'] and if
you simply save it without editing it saves as ['['item1','item2']']
I think the correct would it to reverse parse it to the | syntax, via
widget maybe.
If it was like this it would be simpler to adpat the above mentioned
jQuery plugin.
But the way it is it´s need to do two types of parses.
Is this really wrong or not?
Re: [web2py] Re: Facebook oauth
The fixes have been applied, they are on trunk
http://code.google.com/p/web2py/source/checkout
they should work here is an updated version of the example app running on GAE:
http://grafbook.appspot.com/helloFacebook/graph
if it works it should mantain the url above even after authentication with fb.
mic
2010/8/27 mdipierro mdipie...@cs.depaul.edu:
I think so but I did not try it.
On Aug 26, 11:40 pm, Narendran gunanar...@gmail.com wrote:
Hello Michele/Massimo,
Has this fix been taken in? Is it available in any web2py build now?
On Aug 22, 4:10 am, Michele Comitini michele.comit...@gmail.com
wrote:
Massimo here is the full patch with modifications for
oauth*_account.py login_methods
tnx!
2010/8/22 Michele Comitini michele.comit...@gmail.com:
in tools.py line 1436:
next = URL(r=request) + '?_next=' + next
should be like this?
next = self.url('user', args='login') + '?_next=' + next
2010/8/22 Michele Comitini michele.comit...@gmail.com:
Ok found a way to fix the problem with redirection (_next parameter).
Massimo pls look if the following is correct!
diff -r 9261ce4eda7f gluon/tools.py
--- a/gluon/tools.py Thu Aug 19 04:13:54 2010 +0200
+++ b/gluon/tools.py Sun Aug 22 00:08:25 2010 +0200
@@ -982,7 +983,7 @@
request = self.environment.request
args = request.args
if not args:
- redirect(self.url(args='login'))
+ redirect(self.url(args='login', vars=request.vars))
elif args[0] in self.settings.actions_disabled:
raise HTTP(404)
if args[0] == 'login':
2010/8/20 Michele Comitini michele.comit...@gmail.com:
Hello Narendran,
Do not use that it is old..
now facebook is supported inside web2py distribution with
oauth20_account.py
you can find an example app here:
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
for a simple example usage of graph api look here:
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
http://code.google.com/r/michelecomitini-facebookaccess/source/browse...
for the redirection after login I am investigating...
mic
2010/8/20 Narendran gunanar...@gmail.com:
Hello all,
I picked the Facebook oauth submitted by mcm from
https://code.google.com/r/michelecomitini-facebookaccess/source/brows...
(referred in this thread:
http://groups.google.com/group/web2py/browse_thread/thread/be441047bf...)
I am facing one isse:
1. I've placed require_login decoration on a method say a(). After the
authentication is complete, the page always gets redirected to
default/
index, whereas the ideal behaviour would be to go to a(). It works as
intended if I use default auth instead of Facebook auth.
Also, is there any plugin/package that would provide full-fledged
support for using Facebook Graph API with web2py?
--
Thanks
Narendran
oauth_action_flow_fix.diff
17KViewDownload
[web2py] Re: handing a file upload submitted outside of web2py
thanks for replying.
It's not an image that's being uploaded but instead an XML file which
I will be parsing the contents of into a database table.
Carl
On Aug 27, 5:24 pm, Martin.Mulone mulone.mar...@gmail.com wrote:
do you want to access to the image uploaded with form for example?.
use download function in controller
img src={{=URL('download', args=item.image)}}
width=200px /
On Aug 27, 12:08 pm, Carl carl.ro...@gmail.com wrote:
I'm using Pyjamas to build my client app and Web2py to build the
server.
I'm submitting a HTML form (generated by Pyjamas) to the Web2py server
code.
Where should I look to see how I can handle the submission using
Web2py?
I already have a function defined in /app/controllers/default.py which
gets called with the form data from the web browser (stuff like
request.vars).
The return value from my function is correctly returned to the client
but obviously I need to access the file the user selected and store it
in the database (I'm using GAE so storing in the database is the route
I need to take)
Any pointers?
Re: [web2py] Problem with Oracle connection string...
On 2010-08-27, 15:06 ionel wrote: Hello, I have a problem when I try to connect to an Oracle database with this connection string: db = DAL(oracle://username:password@//network_server:1521/ database_name) Hello, Replacing : with / works for me on Oracle 11.0.2. Bye. M
Re: [web2py] Problem with Oracle connection string...
On 2010-08-27, 15:06 ionel wrote: Hello, I have a problem when I try to connect to an Oracle database with this connection string: db = DAL(oracle://username:password@//network_server:1521/ database_name) Hello, Replacing : with / in username:password works for me on Oracle 11.0.2. Bye. M
Re: [web2py] Problem with Oracle connection string...
Marcin, did you connect to a remote server? which : did you replace with / ? there is someone else which can test with other versions of oracle? 2010/8/27 Marcin Jaworski marcin.krzysztof.jawor...@gmail.com On 2010-08-27, 15:06 ionel wrote: Hello, I have a problem when I try to connect to an Oracle database with this connection string: db = DAL(oracle://username:password@//network_server:1521/ database_name) Hello, Replacing : with / works for me on Oracle 11.0.2. Bye. M -- http://rochacbruno.com.br
[web2py] Bug or Feature , cannot store DB in session..
I am trying to store dynamically generated DB into session but it fails with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
Re: [web2py] Re: Trying to multiprocess
strings is very neat unix/linux command to extract Strings (with more than 4 chars by default) inside any type of files (even binary files, images , etc). so if there a python implemantion of it , if u know i will use it. as Kevin shows theres not much speed difference in IO compare to C. (even faster hmmm , but not every case i guess) . If not as michel suggested , i will fork. But forking inside web2py , will it work? You mean outside of web2py ? it will need IPC/Socks to communicate between , well i can do it in twisted but that really necessary? Oh another thing , the indexer (as soon as index is finished , it just put inside db and only repond with done) so your suggestion , to make master process and to poll will work. On Fri, Aug 27, 2010 at 4:08 AM, Michele Comitini michele.comit...@gmail.com wrote: Phyo, I agree mostly with what Kevin says, but some multiprocessing could be good for the case, unless strings is faster than IO. Since the specific problem is not web specific, I suggest that you do create a program in python (without web2py) and, as Kevin says, better if you replace strings with some python library function (if it is possible). The program should handle with a master process, and as Massimo suggests, its pool of parallel children tasks (I suggest fork if you are on linux for sure), probably no more than the number of cpus/cores available, you must tune that. The master process should be able to respond to status request and eventually handle a clean shutdown through some handle. Once you have your plain python program functioning, make it a module and import it in a web2py controller. You should then be able to poll the master process (fork and keep the hanlde reference in session) in a web2py controller so you can use a nice jquery widget to show progress. :-) mic 2010/8/26 Kevin extemporalgen...@gmail.com: Although there are many places where multiprocess'ing could be handy and efficient, unfortunately string matching is one of those things that is almost entirely I/O bound. With I/O bound tasks (particularly the kind of processing you showed in your example code), you'll be spending over 90% of your time waiting for the disk to supply the data. A couple of characteristics of these kinds of tasks: * You will get essentially zero tangible total performance improvement if you have a single hard drive whether you're running single threaded on a single processor, or 500,000 processes on a super-computer -- it'll all get completed in about the same number of seconds either way (probably saving a little time going single-threaded). * On python, I/O bound tasks complete in about the same amount of time as the equivalent code written in pure ANSI C (see http://www.pytips.com/2010/5/29/a-quick-md5sum-equivalent-in-python -- take the exact timings there with a grain of salt, but it's a pretty good real-world example of what you'll see). So what I would do in your exact situation is to make the equivalent to strings in pure python (the overhead of calling an external process many times definitely will be noticeable), and instead just do it with at most 2 threads (I would go single threaded and only notice about an estimated 2% increase in the total time required to complete all processing). On Aug 20, 6:01 am, Phyo Arkar phyo.arkarl...@gmail.com wrote: well lets say i have about a thounsand files to be proccessed .. i need to extract text out of them , whatever file type it is (i use Linux strings) command . i want to do in multi processed way , which works on multi-core pcs too. this is my current implementation : import subprocess,shlex def __forcedParsing(fname): cmd = 'strings %s' % (fname) #print cmd args= shlex.split(cmd) try: sp = subprocess.Popen( args, shell = False, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) out, err = sp.communicate() except OSError: print Error no %s Message %s % (OSError.errno,OSError.message) pass if sp.returncode== 0: #print Processed %s %fname return out def parseDocs(): rows_to_parse = [i for i in range( 0,len(SESSION.all_docs))] row_ids = [x[0] for x in SESSION.all_docs ] res=[] for rowID in rows_to_parse: file_id, fname, ftype, dir = SESSION.all_docs[int( rowID ) ] fp = os.path.join( dir, fname ) res.append(__forcedParsing(fp)) well the problem is i need output from subprocess so i have to read using sp.communicate(). i need that to be multiprocessed (via forking? poll?) so here are my thoughs : 1) without using fork() , could I do multiple ajax posts by iterating the huge list of files at client side to server , each processes will be
Re: [web2py] Problem with Oracle connection string...
On 2010-08-27, 19:50 Bruno Rocha wrote: Marcin, did you connect to a remote server? which : did you replace with / ? there is someone else which can test with other versions of oracle? Hello, I'm using local connection. Colon was replaced with / in username:password string. Without replacement it did not work at all. Bye. M 2010/8/27 Marcin Jaworski marcin.krzysztof.jawor...@gmail.com On 2010-08-27, 15:06 ionel wrote: Hello, I have a problem when I try to connect to an Oracle database with this connection string: db = DAL(oracle://username:password@//network_server:1521/ database_name) Hello, Replacing : with / works for me on Oracle 11.0.2. Bye. M -- http://rochacbruno.com.br
[web2py] Re: Bug or Feature , cannot store DB in session..
not possible. A db contains an open socket. It cannot be serialized.
On Aug 27, 12:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
I am trying to store dynamically generated DB into session but it fails with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
[web2py] Re: Bug or Feature , cannot store DB in session..
It is not possible because a DB contains a database connection. That
cannot be serialized.
On Aug 27, 12:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
I am trying to store dynamically generated DB into session but it fails with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
Re: [web2py] Re: Bug or Feature , cannot store DB in session..
ok i c so theres no possible way to share ? so how the dbs inside models
works? they have to re-execute every time requested?
On Sat, Aug 28, 2010 at 12:57 AM, mdipierro mdipie...@cs.depaul.edu wrote:
not possible. A db contains an open socket. It cannot be serialized.
On Aug 27, 12:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
I am trying to store dynamically generated DB into session but it fails
with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
[web2py] Re: plugin legacy mysql: generates web2py code to access your mysql legacy db
Excellent! I was just coming up on a personal project that requires me to use a legacy database. I wasn't looking forward to writing the models myself. Thanks for this plugin. Dalen On Aug 26, 5:36 pm, selecta gr...@delarue-berlin.de wrote: Create the web2py code needed to access your mysql legacy db. To make this work all the legacy tables you want to access need to have an id field. This plugin needs: mysql mysqldump installed and globally available. Under Windows you will probably need to add the mysql executable directory to the PATH variable, you will also need to modify mysql to mysql.exe and mysqldump to mysqldump.exe. Just guessing here :) Access your tables with: legacy_db(legacy_db.mytable.id0).select() If the script crashes this is might be due to that fact that the data_type_map dictionary is incomplete. Please complete it, improve it and continue. downloadhttp://jaguar.biologie.hu-berlin.de/~fkrause/web2py.plugin.legacymysq... screenshothttp://jaguar.biologie.hu-berlin.de/~fkrause/screenshot_legacymysql.png
[web2py] uWSGI examples page updated for web2py
Hi all, we have added a section in our examples wiki page for web2py: http://projects.unbit.it/uwsgi/wiki/Example I hope this can be useful -- Roberto De Ioris http://unbit.it JID: robe...@jabber.unbit.it
Re: [web2py] Re: Bug or Feature , cannot store DB in session..
thanks massimo
so now i try to re execute :
case_db=DAL('mysql://root@
localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
on every controller of every function which needs it.
On Sat, Aug 28, 2010 at 12:56 AM, mdipierro mdipie...@cs.depaul.edu wrote:
It is not possible because a DB contains a database connection. That
cannot be serialized.
On Aug 27, 12:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
I am trying to store dynamically generated DB into session but it fails
with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
[web2py] Table migration problems with Sqlite
Hi everyone, this is a long one
today I noticed one of my _settings.table file was written
everytime. The table definition was changed a long ago, from the old
chema:
CREATE TABLE settings(
id INTEGER PRIMARY KEY AUTOINCREMENT,
key CHAR(512),
value CHAR(512)
);
to the new:
CREATE TABLE settings(
id INTEGER PRIMARY KEY AUTOINCREMENT,
key CHAR(512) NOT NULL UNIQUE,
value CHAR(512)
);
the change is, key column was made NOT NULL UNIQUE.
So first question I noticed. In the migrate procedure, I noticed the
following code:
for key in keys:
if not key in sql_fields_old:
query = ['ALTER TABLE %s ADD %s %s;' % \
(self._tablename, key,
sql_fields_aux[key].replace(', ', new_add))]
elif self._db._dbname == 'sqlite':
query = None
...
so, for SQL databases, if a column is removed or dropped, it's not
migrated for sqlite. Is there a reason for this?
Then I tried commenting the if self._db._dbname == 'sqlite': line,
and I got the following SQL error:
OperationalError: Cannot add a UNIQUE column
which makes sense, it's not trivial to add a UNIQUE column to an
existing database...
But what makes me worry is, at the end of the _migrate method, I find this:
if query:
...
if key in sql_fields:
sql_fields_old[key] = sql_fields[key]
else:
del sql_fields_old[key]
tfile = open(self._dbt, 'w')
portalocker.lock(tfile, portalocker.LOCK_EX)
print Here2:, self._dbt, sql_fields, sql_fields_old
cPickle.dump(sql_fields_old, tfile)
portalocker.unlock(tfile)
tfile.close()
First, sql_fields_old is updated with the migrated value ONLY if query
is not None. For sqlite it's None for changed columns. So, later,
cPickle dumps the value of sql_fields_old, so the file
_settings.table is written with the same old values, as this field
was not migrated.
So, for every request, web2py detects a migration is needed, but it
does nothing but writing the .table file with the old values.
I think the correct behaviour should be throwing an error if migration
can't be done.
Can you suggest a fix for this?
Thanks very much.
--
Álvaro J. Iradier Muro
Departamento de Desarrollo
alvaro.irad...@polartech.es
[web2py] Re: uWSGI examples page updated for web2py
- (please do not get too excited about the next lines) - no way ;-) tks. Niphlod
[web2py] Re: plugin legacy mysql: generates web2py code to access your mysql legacy db
Nice! On Aug 26, 5:36 pm, selecta gr...@delarue-berlin.de wrote: Create the web2py code needed to access your mysql legacy db. To make this work all the legacy tables you want to access need to have an id field. This plugin needs: mysql mysqldump installed and globally available. Under Windows you will probably need to add the mysql executable directory to the PATH variable, you will also need to modify mysql to mysql.exe and mysqldump to mysqldump.exe. Just guessing here :) Access your tables with: legacy_db(legacy_db.mytable.id0).select() If the script crashes this is might be due to that fact that the data_type_map dictionary is incomplete. Please complete it, improve it and continue. downloadhttp://jaguar.biologie.hu-berlin.de/~fkrause/web2py.plugin.legacymysq... screenshothttp://jaguar.biologie.hu-berlin.de/~fkrause/screenshot_legacymysql.png
Re: [web2py] Re: Trying to multiprocess
2010/8/27 Phyo Arkar phyo.arkarl...@gmail.com: strings is very neat unix/linux command to extract Strings (with more than 4 chars by default) inside any type of files (even binary files, images , etc). so if there a python implemantion of it , if u know i will use it. as Kevin shows theres not much speed difference in IO compare to C. (even faster hmmm , but not every case i guess) . I know strings, very useful ;-). If not as michel suggested , i will fork. But forking inside web2py , will it work? You mean outside of web2py ? it will need IPC/Socks to communicate between , well i can do it in twisted but that really necessary? Yes I mean forking which means the process goes out of web2py flow of control. twisted may be an option, but also below and choose what sounds better: http://wiki.python.org/moin/ParallelProcessing Forget about java threading stuff, you will see that it is much easier to do parallel programming with Python, enjoy! :-) Oh another thing , the indexer (as soon as index is finished , it just put inside db and only repond with done) so your suggestion , to make master process and to poll will work. You can make a list of files to be processed, compute the total size and have the master process report percent of job done... Do not know what kind of files you are indexing maybe look at these: http://arcvback.com/python-indexing.html On Fri, Aug 27, 2010 at 4:08 AM, Michele Comitini michele.comit...@gmail.com wrote: Phyo, I agree mostly with what Kevin says, but some multiprocessing could be good for the case, unless strings is faster than IO. Since the specific problem is not web specific, I suggest that you do create a program in python (without web2py) and, as Kevin says, better if you replace strings with some python library function (if it is possible). The program should handle with a master process, and as Massimo suggests, its pool of parallel children tasks (I suggest fork if you are on linux for sure), probably no more than the number of cpus/cores available, you must tune that. The master process should be able to respond to status request and eventually handle a clean shutdown through some handle. Once you have your plain python program functioning, make it a module and import it in a web2py controller. You should then be able to poll the master process (fork and keep the hanlde reference in session) in a web2py controller so you can use a nice jquery widget to show progress. :-) mic 2010/8/26 Kevin extemporalgen...@gmail.com: Although there are many places where multiprocess'ing could be handy and efficient, unfortunately string matching is one of those things that is almost entirely I/O bound. With I/O bound tasks (particularly the kind of processing you showed in your example code), you'll be spending over 90% of your time waiting for the disk to supply the data. A couple of characteristics of these kinds of tasks: * You will get essentially zero tangible total performance improvement if you have a single hard drive whether you're running single threaded on a single processor, or 500,000 processes on a super-computer -- it'll all get completed in about the same number of seconds either way (probably saving a little time going single-threaded). * On python, I/O bound tasks complete in about the same amount of time as the equivalent code written in pure ANSI C (see http://www.pytips.com/2010/5/29/a-quick-md5sum-equivalent-in-python -- take the exact timings there with a grain of salt, but it's a pretty good real-world example of what you'll see). So what I would do in your exact situation is to make the equivalent to strings in pure python (the overhead of calling an external process many times definitely will be noticeable), and instead just do it with at most 2 threads (I would go single threaded and only notice about an estimated 2% increase in the total time required to complete all processing). On Aug 20, 6:01 am, Phyo Arkar phyo.arkarl...@gmail.com wrote: well lets say i have about a thounsand files to be proccessed .. i need to extract text out of them , whatever file type it is (i use Linux strings) command . i want to do in multi processed way , which works on multi-core pcs too. this is my current implementation : import subprocess,shlex def __forcedParsing(fname): cmd = 'strings %s' % (fname) #print cmd args= shlex.split(cmd) try: sp = subprocess.Popen( args, shell = False, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) out, err = sp.communicate() except OSError: print Error no %s Message %s % (OSError.errno,OSError.message) pass if sp.returncode== 0: #print Processed %s %fname return out def parseDocs(): rows_to_parse = [i for i in
[web2py] ANN: first pre-alfa release (v0.2) of Web2Py_CC
hello, I'm proud to present the first pre-alfa release of Web2Py_CC. *Windows*: as it's developed under windows, it runs without exceptions (and of course a few bugs ;-) Sorry, I wasn't able to produce an executable, because py2exe had problems (probably with my recently upgraded wxPython, and I can't go back). *Fedora 13*: runs as expected *Ubuntu 10*: 2 days ago, it runned as expected, but after an adviced upgrade of wxPython, my complete Ubuntu installation is ruined. *Mac*: sorry I've no access to Mac you can find some documentation here: http://mientki.ruhosting.nl/web2py/web2py_cc.html a download link and installation instructions are at the bottom of the above page. have fun, Stef
[web2py] Some questions about embedding debugging ...
hello, I'm looking if I can embed debugging in Web2Py_CC, but I've a few questions before I start my attempts. 1. What is the allowed / preferred debugger, pdb or winpdb ? 2. A run some tests in a debugger (winpdb) and saw that exceptions are caught always in web2py (which might be a very logical choice). Is it possible to let the exceptions be handled by an external debugger ? 3. Is it possible to redirect the output so print statements can be seen ? (btw where are print statements going now ?) thanks, Stef Mientki
[web2py] Re: ANN: first pre-alfa release (v0.2) of Web2Py_CC
I ran it, got error D:\Development\python\web2py_cc\Web2Pypython Web2py_CC.py Traceback (most recent call last): File Web2py_CC.py, line 15, in module ] TypeError: list indices must be integers, not tuple Pai On Aug 27, 3:13 pm, Stef Mientki stef.mien...@gmail.com wrote: hello, I'm proud to present the first pre-alfa release of Web2Py_CC. *Windows*: as it's developed under windows, it runs without exceptions (and of course a few bugs ;-) Sorry, I wasn't able to produce an executable, because py2exe had problems (probably with my recently upgraded wxPython, and I can't go back). *Fedora 13*: runs as expected *Ubuntu 10*: 2 days ago, it runned as expected, but after an adviced upgrade of wxPython, my complete Ubuntu installation is ruined. *Mac*: sorry I've no access to Mac you can find some documentation here: http://mientki.ruhosting.nl/web2py/web2py_cc.html a download link and installation instructions are at the bottom of the above page. have fun, Stef
[web2py] Sqlite string type
Just wondering why the SQLiteAdapter uses CHAR as the format for 'string' instead of VARCHAR like the other adapters? From here- http://sqlite.org/datatype3.html: If the declared type of the column contains any of the strings CHAR, CLOB, or TEXT then that column has TEXT affinity. Notice that the type VARCHAR contains the string CHAR and is thus assigned TEXT affinity. Not a big issue, just would make cross database movements easier. Thanks, -- Adrian Klaver adrian.kla...@gmail.com
Re: [web2py] Re: Trying to multiprocess
Yes i do use forks before in C and Python . But i have not tried with web2py yet , i dont know how web2py will behave . Forks works well on web2py? if not i will just write a twisted base daemon, which will listen on a unix ipc socket for requests and if the crawling is done , will report number_of_processed_files back to web2py. it can just call back directly to a web2py controller's function which waits for it using urlib. Do not know what kind of files you are indexing maybe look at these: http://arcvback.com/python-indexing.html what i am doing now is trying to parse any kind of files. well lets say it is a crawler (not and indexer) yet. it crawls and insert into database. for indexer i already using sphinx. Fastest one ever,from my tests. On Sat, Aug 28, 2010 at 2:35 AM, Michele Comitini michele.comit...@gmail.com wrote: 2010/8/27 Phyo Arkar phyo.arkarl...@gmail.com: strings is very neat unix/linux command to extract Strings (with more than 4 chars by default) inside any type of files (even binary files, images , etc). so if there a python implemantion of it , if u know i will use it. as Kevin shows theres not much speed difference in IO compare to C. (even faster hmmm , but not every case i guess) . I know strings, very useful ;-). If not as michel suggested , i will fork. But forking inside web2py , will it work? You mean outside of web2py ? it will need IPC/Socks to communicate between , well i can do it in twisted but that really necessary? Yes I mean forking which means the process goes out of web2py flow of control. twisted may be an option, but also below and choose what sounds better: http://wiki.python.org/moin/ParallelProcessing Forget about java threading stuff, you will see that it is much easier to do parallel programming with Python, enjoy! :-) Oh another thing , the indexer (as soon as index is finished , it just put inside db and only repond with done) so your suggestion , to make master process and to poll will work. You can make a list of files to be processed, compute the total size and have the master process report percent of job done... Do not know what kind of files you are indexing maybe look at these: http://arcvback.com/python-indexing.html On Fri, Aug 27, 2010 at 4:08 AM, Michele Comitini michele.comit...@gmail.com wrote: Phyo, I agree mostly with what Kevin says, but some multiprocessing could be good for the case, unless strings is faster than IO. Since the specific problem is not web specific, I suggest that you do create a program in python (without web2py) and, as Kevin says, better if you replace strings with some python library function (if it is possible). The program should handle with a master process, and as Massimo suggests, its pool of parallel children tasks (I suggest fork if you are on linux for sure), probably no more than the number of cpus/cores available, you must tune that. The master process should be able to respond to status request and eventually handle a clean shutdown through some handle. Once you have your plain python program functioning, make it a module and import it in a web2py controller. You should then be able to poll the master process (fork and keep the hanlde reference in session) in a web2py controller so you can use a nice jquery widget to show progress. :-) mic 2010/8/26 Kevin extemporalgen...@gmail.com: Although there are many places where multiprocess'ing could be handy and efficient, unfortunately string matching is one of those things that is almost entirely I/O bound. With I/O bound tasks (particularly the kind of processing you showed in your example code), you'll be spending over 90% of your time waiting for the disk to supply the data. A couple of characteristics of these kinds of tasks: * You will get essentially zero tangible total performance improvement if you have a single hard drive whether you're running single threaded on a single processor, or 500,000 processes on a super-computer -- it'll all get completed in about the same number of seconds either way (probably saving a little time going single-threaded). * On python, I/O bound tasks complete in about the same amount of time as the equivalent code written in pure ANSI C (see http://www.pytips.com/2010/5/29/a-quick-md5sum-equivalent-in-python-- take the exact timings there with a grain of salt, but it's a pretty good real-world example of what you'll see). So what I would do in your exact situation is to make the equivalent to strings in pure python (the overhead of calling an external process many times definitely will be noticeable), and instead just do it with at most 2 threads (I would go single threaded and only notice about an estimated 2% increase in the total time required to complete all processing). On Aug 20, 6:01 am, Phyo Arkar
[web2py] Re: How may I revoked a user logged in server side
I have a similar but perhaps simpler problem. When I detect excessive activity that might be a bot, I want to 1. Log the user out 2. Force a re-captcha on the next login only. So if I have something like: try: if form.accepts(...): etc ... except ActivityLimitException: session.flash(Whoa there, pardner!) ## What should go here? Thanks, Mike On Aug 26, 5:09 pm, Jean-Guy jean...@gmail.com wrote: Ok! It clear now. Yes could be a nice to have... But Web2py is already a wonderfully nice framework ;-) Ciao! Jonhy On 2010-08-26 16:17, mdipierro wrote: e the session file. We do not have an utility for it. We should make one.
Re: [web2py] Re: Trying to multiprocess
2010/8/27 Phyo Arkar phyo.arkarl...@gmail.com: Yes i do use forks before in C and Python . Forks works well on web2py? From my experience I would say so. what i am doing now is trying to parse any kind of files. well lets say it is a crawler (not and indexer) yet. it crawls and insert into database. for indexer i already using sphinx. Fastest one ever,from my tests. do you use postgresql? if so why not using this? http://www.postgresql.org/docs/8.4/interactive/textsearch.html it is fast!
Re: [web2py] Re: Trying to multiprocess
Ok then i should start trying fork on web2py , thx michele. On Sat, Aug 28, 2010 at 3:32 AM, Michele Comitini michele.comit...@gmail.com wrote: 2010/8/27 Phyo Arkar phyo.arkarl...@gmail.com: Yes i do use forks before in C and Python . Forks works well on web2py? From my experience I would say so. what i am doing now is trying to parse any kind of files. well lets say it is a crawler (not and indexer) yet. it crawls and insert into database. for indexer i already using sphinx. Fastest one ever,from my tests. do you use postgresql? if so why not using this? http://www.postgresql.org/docs/8.4/interactive/textsearch.html it is fast!
Re: [web2py] Re: Bug or Feature , cannot store DB in session = impossible [Solved]
Closed :)
On Sat, Aug 28, 2010 at 1:24 AM, Phyo Arkar phyo.arkarl...@gmail.comwrote:
thanks massimo
so now i try to re execute :
case_db=DAL('mysql://root@
localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
on every controller of every function which needs it.
On Sat, Aug 28, 2010 at 12:56 AM, mdipierro mdipie...@cs.depaul.eduwrote:
It is not possible because a DB contains a database connection. That
cannot be serialized.
On Aug 27, 12:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
I am trying to store dynamically generated DB into session but it fails
with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
Re: [web2py] Re: Trying to multiprocess
Both MySQL and Postgre have index searching , yes. I have not tried PostgreSQL FTS but SphinxSearch is renowned for fastest indexing and fastest search. its python api is very easy too, you should give it a try. http://sphinxsearch.com/ On Sat, Aug 28, 2010 at 3:32 AM, Michele Comitini michele.comit...@gmail.com wrote: 2010/8/27 Phyo Arkar phyo.arkarl...@gmail.com: Yes i do use forks before in C and Python . Forks works well on web2py? From my experience I would say so. what i am doing now is trying to parse any kind of files. well lets say it is a crawler (not and indexer) yet. it crawls and insert into database. for indexer i already using sphinx. Fastest one ever,from my tests. do you use postgresql? if so why not using this? http://www.postgresql.org/docs/8.4/interactive/textsearch.html it is fast!
[web2py] Re: Bug or Feature , cannot store DB in session..
Can you explain your design?
On Aug 27, 1:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
thanks massimo
so now i try to re execute :
case_db=DAL('mysql://root@
localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
on every controller of every function which needs it.
On Sat, Aug 28, 2010 at 12:56 AM, mdipierro mdipie...@cs.depaul.edu wrote:
It is not possible because a DB contains a database connection. That
cannot be serialized.
On Aug 27, 12:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
I am trying to store dynamically generated DB into session but it fails
with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
[web2py] Re: uWSGI examples page updated for web2py
thanks. Some users have experience dropped requests with uWSGI on heavy traffic. Can you reproduce the problem? On Aug 26, 11:32 pm, Roberto De Ioris robe...@unbit.it wrote: Hi all, we have added a section in our examples wiki page for web2py: http://projects.unbit.it/uwsgi/wiki/Example I hope this can be useful -- Roberto De Iorishttp://unbit.it JID: robe...@jabber.unbit.it
[web2py] Re: Table migration problems with Sqlite
so, for SQL databases, if a column is removed or dropped, it's not
migrated for sqlite. Is there a reason for this?
sqlite does not support drop columns.
On Aug 27, 2:03 pm, Álvaro J. Iradier alvaro.irad...@polartech.es
wrote:
Hi everyone, this is a long one
today I noticed one of my _settings.table file was written
everytime. The table definition was changed a long ago, from the old
chema:
CREATE TABLE settings(
id INTEGER PRIMARY KEY AUTOINCREMENT,
key CHAR(512),
value CHAR(512)
);
to the new:
CREATE TABLE settings(
id INTEGER PRIMARY KEY AUTOINCREMENT,
key CHAR(512) NOT NULL UNIQUE,
value CHAR(512)
);
the change is, key column was made NOT NULL UNIQUE.
So first question I noticed. In the migrate procedure, I noticed the
following code:
for key in keys:
if not key in sql_fields_old:
query = ['ALTER TABLE %s ADD %s %s;' % \
(self._tablename, key,
sql_fields_aux[key].replace(', ', new_add))]
elif self._db._dbname == 'sqlite':
query = None
...
so, for SQL databases, if a column is removed or dropped, it's not
migrated for sqlite. Is there a reason for this?
Then I tried commenting the if self._db._dbname == 'sqlite': line,
and I got the following SQL error:
OperationalError: Cannot add a UNIQUE column
which makes sense, it's not trivial to add a UNIQUE column to an
existing database...
But what makes me worry is, at the end of the _migrate method, I find this:
if query:
...
if key in sql_fields:
sql_fields_old[key] = sql_fields[key]
else:
del sql_fields_old[key]
tfile = open(self._dbt, 'w')
portalocker.lock(tfile, portalocker.LOCK_EX)
print Here2:, self._dbt, sql_fields, sql_fields_old
cPickle.dump(sql_fields_old, tfile)
portalocker.unlock(tfile)
tfile.close()
First, sql_fields_old is updated with the migrated value ONLY if query
is not None. For sqlite it's None for changed columns. So, later,
cPickle dumps the value of sql_fields_old, so the file
_settings.table is written with the same old values, as this field
was not migrated.
So, for every request, web2py detects a migration is needed, but it
does nothing but writing the .table file with the old values.
I think the correct behaviour should be throwing an error if migration
can't be done.
Can you suggest a fix for this?
Thanks very much.
--
Álvaro J. Iradier Muro
Departamento de Desarrollo
alvaro.irad...@polartech.es
Re: [web2py] Re: Trying to multiprocess
its python api is very easy too, you should give it a try. http://sphinxsearch.com/ yes I will
Re: [web2py] Re: Bug or Feature , cannot store DB in session..
Here is how it works:
At Home page , theres a list of DBs , lets say
Db Name , Owner , Description
Db1 , JohnSmith, Test
Db2 , JaneSmith, Test
Db3 , JohnDoe, Test
in jqgrid.
User uploads files as an archive then clicks on Db1 and click start
processing.(note i have to use Db directly, clients saids they are easier to
manage by dbs not tables).
when start processing clicked , Db1 is posted to function named
extraction/index, there under extraction/index archive is extracted and
crawled , then resulting text are inserted into db.
db is called as below for every action it need databases:
def index():
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(file_data(...))
crawdata=Crawler(path)
case_db.file_data.insert(crawdata)
after extraction is done , it stores by inserting the crawled Data inside
the case_db.
On Next page , it is results function which gives data result view , from
parsed docs.
there also have to call the db on every ajax request inside jqgrid as below
def rows_ajax()
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
res=case_db(query).select()
return (dict(res=res)
so what i want to do is instead of calling db everytime , i want to store it
inside somewhere accessible globally ..
On Sat, Aug 28, 2010 at 4:00 AM, mdipierro mdipie...@cs.depaul.edu wrote:
Can you explain your design?
On Aug 27, 1:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
thanks massimo
so now i try to re execute :
case_db=DAL('mysql://root@
localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
on every controller of every function which needs it.
On Sat, Aug 28, 2010 at 12:56 AM, mdipierro mdipie...@cs.depaul.edu
wrote:
It is not possible because a DB contains a database connection. That
cannot be serialized.
On Aug 27, 12:54 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote:
I am trying to store dynamically generated DB into session but it
fails
with
error . is that supported or if i want to share DB Globally , across
controller , only within a session, but it is dynamically generated
how
should i share without puttint into models??
case_db=DAL('mysql://r...@localhost/'+ request.vars.db_name)
case_db.define_table(...)
session.case_db=case_db
Traceback (most recent call last):
File /home/v3ss/workspace-bbb/web2py-clone/gluon/main.py, line
411, in wsgibase
session._try_store_on_disk(request, response)
File /home/v3ss/workspace-bbb/web2py-clone/gluon/globals.py, line
377, in _try_store_on_disk
cPickle.dump(dict(self), response.session_file)
File /usr/lib/python2.6/copy_reg.py, line 74, in _reduce_ex
getstate = self.__getstate__
File /home/v3ss/workspace-bbb/web2py-clone/gluon/sql.py, line
1380, in __getattr__
return dict.__getitem__(self,key)
KeyError: '__getstate__'
Re: [web2py] Re: ANN: first pre-alfa release (v0.2) of Web2Py_CC
On 27-08-2010 22:32, Pai wrote: I ran it, got error D:\Development\python\web2py_cc\Web2Pypython Web2py_CC.py Traceback (most recent call last): File Web2py_CC.py, line 15, in module ] TypeError: list indices must be integers, not tuple sorry I uploaded the wrong file. cheers, Stef Pai On Aug 27, 3:13 pm, Stef Mientki stef.mien...@gmail.com wrote: hello, I'm proud to present the first pre-alfa release of Web2Py_CC. *Windows*: as it's developed under windows, it runs without exceptions (and of course a few bugs ;-) Sorry, I wasn't able to produce an executable, because py2exe had problems (probably with my recently upgraded wxPython, and I can't go back). *Fedora 13*: runs as expected *Ubuntu 10*: 2 days ago, it runned as expected, but after an adviced upgrade of wxPython, my complete Ubuntu installation is ruined. *Mac*: sorry I've no access to Mac you can find some documentation here: http://mientki.ruhosting.nl/web2py/web2py_cc.html a download link and installation instructions are at the bottom of the above page. have fun, Stef
[web2py] Field default
I seem to be having problems getting my head around the default argument to
Field.
I am using SQLite as the backend.
If I do:
Field('grade_date','date',default=request.now,required=True,notnull=True)
I get a database column with a DEFAULT of todays date, not what I want:
grade_date DATE NOT NULL DEFAULT '2010-08-27'
If I do not include the default argument in the initial define_table but add
it
later todays date shows up in the form but there is no DEFAULT added to the
column.
What I would like to do is pass the SQLite date variable CURRENT_DATE to the
table. At this point I am unclear as to whether default is associated with the
table or the form.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
Re: [web2py] Field default
I dont know if I understand clearly what you want, but..
May be :
def get_sql_date():
return db.executesql('select CURRENT_DATE')
Field('grade_date','date',compute=get_sql_date,required=True,notnull=True)
or
Field('grade_date','date',default=request.now.date().strftime('%Y-%m-%d'),required=True,notnull=True)
2010/8/27 Adrian Klaver adrian.kla...@gmail.com
I seem to be having problems getting my head around the default argument to
Field.
I am using SQLite as the backend.
If I do:
Field('grade_date','date',default=request.now,required=True,notnull=True)
I get a database column with a DEFAULT of todays date, not what I want:
grade_date DATE NOT NULL DEFAULT '2010-08-27'
If I do not include the default argument in the initial define_table but
add it
later todays date shows up in the form but there is no DEFAULT added to the
column.
What I would like to do is pass the SQLite date variable CURRENT_DATE to
the
table. At this point I am unclear as to whether default is associated with
the
table or the form.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
--
http://rochacbruno.com.br
Re: [web2py] Field default
OPs..
you have to parse the executesql() return, replace with that
db.executesql('select CURRENT_DATE')[0][0].encode()
2010/8/27 Bruno Rocha rochacbr...@gmail.com
I dont know if I understand clearly what you want, but..
May be :
def get_sql_date():
return db.executesql('select CURRENT_DATE')
Field('grade_date','date',compute=get_sql_date,required=True,notnull=True)
or
Field('grade_date','date',default=request.now.date().strftime('%Y-%m-%d'),required=True,notnull=True)
2010/8/27 Adrian Klaver adrian.kla...@gmail.com
I seem to be having problems getting my head around the default argument to
Field.
I am using SQLite as the backend.
If I do:
Field('grade_date','date',default=request.now,required=True,notnull=True)
I get a database column with a DEFAULT of todays date, not what I want:
grade_date DATE NOT NULL DEFAULT '2010-08-27'
If I do not include the default argument in the initial define_table but
add it
later todays date shows up in the form but there is no DEFAULT added to
the
column.
What I would like to do is pass the SQLite date variable CURRENT_DATE to
the
table. At this point I am unclear as to whether default is associated with
the
table or the form.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
--
http://rochacbruno.com.br
--
http://rochacbruno.com.br
Re: [web2py] Field default
On Friday 27 August 2010 4:15:17 pm Bruno Rocha wrote:
I dont know if I understand clearly what you want, but..
May be :
def get_sql_date():
return db.executesql('select CURRENT_DATE')
Field('grade_date','date',compute=get_sql_date,required=True,notnull=True)
or
Field('grade_date','date',default=request.now.date().strftime('%Y-%m-%d'),r
equired=True,notnull=True)
Thanks but that attacks the problem from the client end of things, not the
database. What I am looking for is a way to supply database specific variables
to the table creation process. I would like to end up with the field in the
database having the definition:
grade_date DATE NOT NULL DEFAULT CURRENT_DATE
This would make the DEFAULT have meaning outside of the framework.
--
Adrian Klaver
adrian.kla...@gmail.com
[web2py] Has anyone used standalone web2py DAL in a project ?
Hi All, I'm going to have to use a MySQL orm in a project in near future. Do you think web2py DAL is good for that purpose regarding scalability and performance ? And what files do I need to import or execute to use only web2py DAL ? Thanks.
Re: [web2py] Field default
Ok, now I got it.
But, I dont think DAL has a method for doing that.
I know just notnull, unique and ondelete that are enforced at the
level of the database.
I think you should use raw SQL for that
db.executesql('ALTER TABLE..;)
2010/8/27 Adrian Klaver adrian.kla...@gmail.com
On Friday 27 August 2010 4:15:17 pm Bruno Rocha wrote:
I dont know if I understand clearly what you want, but..
May be :
def get_sql_date():
return db.executesql('select CURRENT_DATE')
Field('grade_date','date',compute=get_sql_date,required=True,notnull=True)
or
Field('grade_date','date',default=request.now.date().strftime('%Y-%m-%d'),r
equired=True,notnull=True)
Thanks but that attacks the problem from the client end of things, not the
database. What I am looking for is a way to supply database specific
variables
to the table creation process. I would like to end up with the field in the
database having the definition:
grade_date DATE NOT NULL DEFAULT CURRENT_DATE
This would make the DEFAULT have meaning outside of the framework.
--
Adrian Klaver
adrian.kla...@gmail.com
--
http://rochacbruno.com.br
[web2py] Server fails every 3 hour
Hi, I am running a web2py application at my home server, that is published and I have some clients using for accurracy tests. But , every 3 hours+- the server fails with the message Exception in thread Thread-20: Traceback (most recent call last): File /usr/lib/python2.6/threading.py, line 532, in __bootstrap_inner self.run() File /home/bruno/web2py/gluon/newcron.py, line 206, in run shell=self.shell) File /usr/lib/python2.6/subprocess.py, line 633, in __init__ errread, errwrite) File /usr/lib/python2.6/subprocess.py, line 1049, in _execute_child self.pid = os.fork() OSError: [Errno 12] Do not possible to allocate memory I can see newcron.py is the problem there, but I need to use cron, this could be a problem with my OS or machine, or it is a Rocket problem? -- http://rochacbruno.com.br
Re: [web2py] Field default
On Friday 27 August 2010 5:05:40 pm Bruno Rocha wrote:
Ok, now I got it.
But, I dont think DAL has a method for doing that.
I know just notnull, unique and ondelete that are enforced at the
level of the database.
I think you should use raw SQL for that
db.executesql('ALTER TABLE..;)
That is a possibility, though it means the model no longer represents the
reality :. Also in the particular case of SQLite the ALTER TABLE syntax does
not support adding a DEFAULT to an existing column. What it comes down to is
whether I stick with DAL managing everything for the benefit of the migrate
feature or manage the schema in the background myself and give up on migrate.
That is something that remains to be determined.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
Re: [web2py] Field default
may be I am wrong .
Wait for Massimo's response
2010/8/27 Adrian Klaver adrian.kla...@gmail.com
On Friday 27 August 2010 5:05:40 pm Bruno Rocha wrote:
Ok, now I got it.
But, I dont think DAL has a method for doing that.
I know just notnull, unique and ondelete that are enforced at the
level of the database.
I think you should use raw SQL for that
db.executesql('ALTER TABLE..;)
That is a possibility, though it means the model no longer represents the
reality :. Also in the particular case of SQLite the ALTER TABLE syntax
does
not support adding a DEFAULT to an existing column. What it comes down to
is
whether I stick with DAL managing everything for the benefit of the migrate
feature or manage the schema in the background myself and give up on
migrate.
That is something that remains to be determined.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
--
http://rochacbruno.com.br
Re: [web2py] Field default
Looking sql.py I found:
'notnull': 'NOT NULL DEFAULT %(default)s'
If I understand it well, that could be yor solution.
SQL_DIALECTS = {'sqlite': {'boolean': 'CHAR(1)',
'string': 'CHAR(%(length)s)','text': 'TEXT',
'password': 'CHAR(%(length)s)','blob': 'BLOB',
'upload': 'CHAR(%(length)s)','integer': 'INTEGER',
'double': 'DOUBLE','decimal': 'DOUBLE','date': 'DATE',
'time': 'TIME','datetime': 'TIMESTAMP','id':
'INTEGER PRIMARY KEY AUTOINCREMENT','reference': 'INTEGER
REFERENCES %(foreign_key)s ON DELETE %(on_delete_action)s',
'lower': 'LOWER(%(field)s)','upper': 'UPPER(%(field)s)',
'is null': 'IS NULL','is not null': 'IS NOT NULL',
'extract': web2py_extract('%(name)s',%(field)s),'left join':
'LEFT JOIN','random': 'Random()','notnull': 'NOT NULL
DEFAULT %(default)s','substring':
'SUBSTR(%(field)s,%(pos)s,%(length)s)','primarykey': 'PRIMARY
KEY (%s)'},
2010/8/27 Bruno Rocha rochacbr...@gmail.com
may be I am wrong .
Wait for Massimo's response
2010/8/27 Adrian Klaver adrian.kla...@gmail.com
On Friday 27 August 2010 5:05:40 pm Bruno Rocha wrote:
Ok, now I got it.
But, I dont think DAL has a method for doing that.
I know just notnull, unique and ondelete that are enforced at the
level of the database.
I think you should use raw SQL for that
db.executesql('ALTER TABLE..;)
That is a possibility, though it means the model no longer represents the
reality :. Also in the particular case of SQLite the ALTER TABLE syntax
does
not support adding a DEFAULT to an existing column. What it comes down to
is
whether I stick with DAL managing everything for the benefit of the
migrate
feature or manage the schema in the background myself and give up on
migrate.
That is something that remains to be determined.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
--
http://rochacbruno.com.br
--
http://rochacbruno.com.br
Re: [web2py] Has anyone used standalone web2py DAL in a project ?
. DON'T DO IT! The DAL is ment for functional programming... so be careful if you have to use it in a class based system. Plus you will have to have your stand alone application manage the .table files too. Just keep it in mind. In any case... You will need. gluon/LICENSE gluon/sql.py gluon/portalocker.py gluon/utils.py Also, you will NOT need validators.. so edit gluon/sql.py and comment out anything to do with the following from serializers import json - remove this, it conflicts with the real version of simplejson installed from pypi. from http import HTTP -- doubt you will be making any http exceptions import validators -- unless you are somehow using SQLFORM, you dont need this. Be sure to remove all traces of validators from the rest of sql.py Just do a search for validators and remove that code. Yay, you now have a stand alone DAL... see how easy that was to de-couple things?! =) -- Thadeus On Fri, Aug 27, 2010 at 6:34 PM, Sujan Shakya suzan.sha...@gmail.com wrote: Hi All, I'm going to have to use a MySQL orm in a project in near future. Do you think web2py DAL is good for that purpose regarding scalability and performance ? And what files do I need to import or execute to use only web2py DAL ? Thanks.
Re: [web2py] Has anyone used standalone web2py DAL in a project ?
Thanks Thadeu, that will be very useful for me too, I will try to follow your instructions. BTW, I am using DAL for a Py/Gtk Application, besides the migration ( does not works very well ) everything else is running ok. I am going to speak about DAL in PyCon Brasil. (google translated) http://translate.google.com.br/translate?js=yprev=_thl=pt-BRie=UTF-8layout=1eotf=1u=http://www.pythonbrasil.org.br/2010/sobre-o-evento/inscricoes/e82ba44698924532bc0af1256dfa288fsl=pttl=en 2010/8/27 Thadeus Burgess thade...@thadeusb.com . DON'T DO IT! The DAL is ment for functional programming... so be careful if you have to use it in a class based system. Plus you will have to have your stand alone application manage the .table files too. Just keep it in mind. In any case... You will need. gluon/LICENSE gluon/sql.py gluon/portalocker.py gluon/utils.py Also, you will NOT need validators.. so edit gluon/sql.py and comment out anything to do with the following from serializers import json - remove this, it conflicts with the real version of simplejson installed from pypi. from http import HTTP -- doubt you will be making any http exceptions import validators -- unless you are somehow using SQLFORM, you dont need this. Be sure to remove all traces of validators from the rest of sql.py Just do a search for validators and remove that code. Yay, you now have a stand alone DAL... see how easy that was to de-couple things?! =) -- Thadeus On Fri, Aug 27, 2010 at 6:34 PM, Sujan Shakya suzan.sha...@gmail.com wrote: Hi All, I'm going to have to use a MySQL orm in a project in near future. Do you think web2py DAL is good for that purpose regarding scalability and performance ? And what files do I need to import or execute to use only web2py DAL ? Thanks. -- http://rochacbruno.com.br
[web2py] Re: Server fails every 3 hour
Run the web server with -N (no cron) and run a separate backrgound process for cron. Anyway, there is no way to control how much memory cron consumes if a cron task takes longer than expected. It is safer not to use cron (-N) and use this instead: http://www.web2py.com/book/default/chapter/04#Background-Processes-and-Task-Queues Massimo On Aug 27, 7:18 pm, Bruno Rocha rochacbr...@gmail.com wrote: Hi, I am running a web2py application at my home server, that is published and I have some clients using for accurracy tests. But , every 3 hours+- the server fails with the message Exception in thread Thread-20: Traceback (most recent call last): File /usr/lib/python2.6/threading.py, line 532, in __bootstrap_inner self.run() File /home/bruno/web2py/gluon/newcron.py, line 206, in run shell=self.shell) File /usr/lib/python2.6/subprocess.py, line 633, in __init__ errread, errwrite) File /usr/lib/python2.6/subprocess.py, line 1049, in _execute_child self.pid = os.fork() OSError: [Errno 12] Do not possible to allocate memory I can see newcron.py is the problem there, but I need to use cron, this could be a problem with my OS or machine, or it is a Rocket problem? -- http://rochacbruno.com.br
Re: [web2py] Field default
On Friday 27 August 2010 5:42:04 pm Bruno Rocha wrote:
Looking sql.py I found:
'notnull': 'NOT NULL DEFAULT %(default)s'
If I understand it well, that could be yor solution.
SQL_DIALECTS = {'sqlite': {'boolean': 'CHAR(1)',
'string': 'CHAR(%(length)s)','text': 'TEXT',
'password': 'CHAR(%(length)s)','blob': 'BLOB',
'upload': 'CHAR(%(length)s)','integer': 'INTEGER',
'double': 'DOUBLE','decimal': 'DOUBLE','date': 'DATE',
'time': 'TIME','datetime': 'TIMESTAMP','id':
'INTEGER PRIMARY KEY AUTOINCREMENT','reference': 'INTEGER
REFERENCES %(foreign_key)s ON DELETE %(on_delete_action)s',
'lower': 'LOWER(%(field)s)','upper': 'UPPER(%(field)s)',
'is null': 'IS NULL','is not null': 'IS NOT NULL',
'extract': web2py_extract('%(name)s',%(field)s),'left join':
'LEFT JOIN','random': 'Random()','notnull': 'NOT NULL
DEFAULT %(default)s','substring':
'SUBSTR(%(field)s,%(pos)s,%(length)s)','primarykey': 'PRIMARY
KEY (%s)'},
Yes and no. That is how the DEFAULT clause is constructed using the string
passed to the default argument in Field()
ex: Field('grade_date','date',default='2010/8/27',required=True,notnull=True)
becomes;
grade_date DATE NOT NULL DEFAULT '2010-08-27'
The problem is CURRENT_DATE is not a string but an SQL function. If I include
it
as a string then the default becomes the string literal 'CURRENT_DATE'. If I do
not quote it as string then Python complains because it is not a declared
variable.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
Re: [web2py] Re: Server fails every 3 hour
Thanks Massimo. , 2010/8/27 mdipierro mdipie...@cs.depaul.edu Run the web server with -N (no cron) and run a separate backrgound process for cron. Anyway, there is no way to control how much memory cron consumes if a cron task takes longer than expected. It is safer not to use cron (-N) and use this instead: http://www.web2py.com/book/default/chapter/04#Background-Processes-and-Task-Queues Massimo On Aug 27, 7:18 pm, Bruno Rocha rochacbr...@gmail.com wrote: Hi, I am running a web2py application at my home server, that is published and I have some clients using for accurracy tests. But , every 3 hours+- the server fails with the message Exception in thread Thread-20: Traceback (most recent call last): File /usr/lib/python2.6/threading.py, line 532, in __bootstrap_inner self.run() File /home/bruno/web2py/gluon/newcron.py, line 206, in run shell=self.shell) File /usr/lib/python2.6/subprocess.py, line 633, in __init__ errread, errwrite) File /usr/lib/python2.6/subprocess.py, line 1049, in _execute_child self.pid = os.fork() OSError: [Errno 12] Do not possible to allocate memory I can see newcron.py is the problem there, but I need to use cron, this could be a problem with my OS or machine, or it is a Rocket problem? -- http://rochacbruno.com.br -- http://rochacbruno.com.br
[web2py] Re: Field default
try this:
from guon.sql import Expression
db.define_table(.,Field(...,default=Expression('CURRENT_DATE')),)
On Aug 27, 9:19 pm, Adrian Klaver adrian.kla...@gmail.com wrote:
On Friday 27 August 2010 5:42:04 pm Bruno Rocha wrote:
Looking sql.py I found:
'notnull': 'NOT NULL DEFAULT %(default)s'
If I understand it well, that could be yor solution.
SQL_DIALECTS = { 'sqlite': { 'boolean': 'CHAR(1)',
'string': 'CHAR(%(length)s)', 'text': 'TEXT',
'password': 'CHAR(%(length)s)', 'blob': 'BLOB',
'upload': 'CHAR(%(length)s)', 'integer': 'INTEGER',
'double': 'DOUBLE', 'decimal': 'DOUBLE', 'date': 'DATE',
'time': 'TIME', 'datetime': 'TIMESTAMP', 'id':
'INTEGER PRIMARY KEY AUTOINCREMENT', 'reference': 'INTEGER
REFERENCES %(foreign_key)s ON DELETE %(on_delete_action)s',
'lower': 'LOWER(%(field)s)', 'upper': 'UPPER(%(field)s)',
'is null': 'IS NULL', 'is not null': 'IS NOT NULL',
'extract': web2py_extract('%(name)s',%(field)s), 'left join':
'LEFT JOIN', 'random': 'Random()', 'notnull': 'NOT NULL
DEFAULT %(default)s', 'substring':
'SUBSTR(%(field)s,%(pos)s,%(length)s)', 'primarykey': 'PRIMARY
KEY (%s)' },
Yes and no. That is how the DEFAULT clause is constructed using the string
passed to the default argument in Field()
ex: Field('grade_date','date',default='2010/8/27',required=True,notnull=True)
becomes;
grade_date DATE NOT NULL DEFAULT '2010-08-27'
The problem is CURRENT_DATE is not a string but an SQL function. If I include
it
as a string then the default becomes the string literal 'CURRENT_DATE'. If I
do
not quote it as string then Python complains because it is not a declared
variable.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
Re: [web2py] Re: Field default
Cool!
hidden tricks and magics of DAL, that is useful, needs to be on the /book
2010/8/27 mdipierro mdipie...@cs.depaul.edu
try this:
from guon.sql import Expression
db.define_table(.,Field(...,default=Expression('CURRENT_DATE')),)
On Aug 27, 9:19 pm, Adrian Klaver adrian.kla...@gmail.com wrote:
On Friday 27 August 2010 5:42:04 pm Bruno Rocha wrote:
Looking sql.py I found:
'notnull': 'NOT NULL DEFAULT %(default)s'
If I understand it well, that could be yor solution.
SQL_DIALECTS = {'sqlite': {'boolean': 'CHAR(1)',
'string': 'CHAR(%(length)s)','text': 'TEXT',
'password': 'CHAR(%(length)s)','blob': 'BLOB',
'upload': 'CHAR(%(length)s)','integer': 'INTEGER',
'double': 'DOUBLE','decimal': 'DOUBLE','date': 'DATE',
'time': 'TIME','datetime': 'TIMESTAMP','id':
'INTEGER PRIMARY KEY AUTOINCREMENT','reference': 'INTEGER
REFERENCES %(foreign_key)s ON DELETE %(on_delete_action)s',
'lower': 'LOWER(%(field)s)','upper': 'UPPER(%(field)s)',
'is null': 'IS NULL','is not null': 'IS NOT NULL',
'extract': web2py_extract('%(name)s',%(field)s),'left join':
'LEFT JOIN','random': 'Random()','notnull': 'NOT NULL
DEFAULT %(default)s','substring':
'SUBSTR(%(field)s,%(pos)s,%(length)s)','primarykey': 'PRIMARY
KEY (%s)'},
Yes and no. That is how the DEFAULT clause is constructed using the
string
passed to the default argument in Field()
ex:
Field('grade_date','date',default='2010/8/27',required=True,notnull=True)
becomes;
grade_date DATE NOT NULL DEFAULT '2010-08-27'
The problem is CURRENT_DATE is not a string but an SQL function. If I
include it
as a string then the default becomes the string literal 'CURRENT_DATE'.
If I do
not quote it as string then Python complains because it is not a declared
variable.
Thanks,
--
Adrian Klaver
adrian.kla...@gmail.com
--
http://rochacbruno.com.br
Re: [web2py] Re: Field default
On Friday 27 August 2010 7:24:13 pm mdipierro wrote:
try this:
from guon.sql import Expression
For those following along: from gluon.sql import Expression
db.define_table(.,Field(...,default=Expression('CURRENT_DATE')),)
On
I tried the above and got:
Traceback (most recent call last):
File /home/aklaver/software_projects/w2py/web2py/gluon/restricted.py, line
186, in restricted
exec ccode in environment
File
/home/aklaver/software_projects/w2py/web2py/applications/hplc/models/db.py,
line 111, in module
Field('date_default','date',default=Expression('CURRENT_DATE'),required=True,notnull=True),
File /home/aklaver/software_projects/w2py/web2py/gluon/sql.py, line 2746,
in
__init__
self.default = default==None and update or default
File /home/aklaver/software_projects/w2py/web2py/gluon/sql.py, line 2548,
in
__eq__
return Query(self, '=', value)
File /home/aklaver/software_projects/w2py/web2py/gluon/sql.py, line 2972,
in
__init__
left._db._translator['is null'])
AttributeError: 'NoneType' object has no attribute '_translator'
--
Adrian Klaver
adrian.kla...@gmail.com
[web2py] Re: Has anyone used standalone web2py DAL in a project ?
Of course, it is always an option (and never a difficulty) to wrap functional programming concepts in an object oriented wrapper -- that's more a benefit of the functional style than then anything OOP gives to you though. On Aug 27, 7:13 pm, Thadeus Burgess thade...@thadeusb.com wrote: . DON'T DO IT! The DAL is ment for functional programming... so be careful if you have to use it in a class based system. Plus you will have to have your stand alone application manage the .table files too. Just keep it in mind. In any case... You will need. gluon/LICENSE gluon/sql.py gluon/portalocker.py gluon/utils.py Also, you will NOT need validators.. so edit gluon/sql.py and comment out anything to do with the following from serializers import json - remove this, it conflicts with the real version of simplejson installed from pypi. from http import HTTP -- doubt you will be making any http exceptions import validators -- unless you are somehow using SQLFORM, you dont need this. Be sure to remove all traces of validators from the rest of sql.py Just do a search for validators and remove that code. Yay, you now have a stand alone DAL... see how easy that was to de-couple things?! =) -- Thadeus On Fri, Aug 27, 2010 at 6:34 PM, Sujan Shakya suzan.sha...@gmail.com wrote: Hi All, I'm going to have to use a MySQL orm in a project in near future. Do you think web2py DAL is good for that purpose regarding scalability and performance ? And what files do I need to import or execute to use only web2py DAL ? Thanks.
[web2py] Re: Field default
Try
db.define_table(.,Field(...,default=Expression('CURRENT_DATE',db=db)),)
I cannot promise this works but let us know.
On Aug 27, 9:46 pm, Adrian Klaver adrian.kla...@gmail.com wrote:
On Friday 27 August 2010 7:24:13 pm mdipierro wrote:
try this:
from guon.sql import Expression
For those following along: from gluon.sql import Expression
db.define_table(.,Field(...,default=Expression('CURRENT_DATE')),)
On
I tried the above and got:
Traceback (most recent call last):
File /home/aklaver/software_projects/w2py/web2py/gluon/restricted.py, line
186, in restricted
exec ccode in environment
File
/home/aklaver/software_projects/w2py/web2py/applications/hplc/models/db.py,
line 111, in module
Field('date_default','date',default=Expression('CURRENT_DATE'),required=True,notnull=True),
File /home/aklaver/software_projects/w2py/web2py/gluon/sql.py, line 2746,
in
__init__
self.default = default==None and update or default
File /home/aklaver/software_projects/w2py/web2py/gluon/sql.py, line 2548,
in
__eq__
return Query(self, '=', value)
File /home/aklaver/software_projects/w2py/web2py/gluon/sql.py, line 2972,
in
__init__
left._db._translator['is null'])
AttributeError: 'NoneType' object has no attribute '_translator'
--
Adrian Klaver
adrian.kla...@gmail.com
Re: [web2py] Re: Has anyone used standalone web2py DAL in a project ?
I use web2py's DAL in my project (daemon). I modified db.py so that all calls to create_table happen with migrate=False (because I reuse the same db as in web app). So all I need from DAL is .select, .update, .delete and .insert. Also I take care to call these functions only from single thread. That's all and it works for me. Ah, yes, the files: in the current folder I have symlinks to gluon and to db.py Also in db.py I have to explicitly check for DAL presence and if missing - import it from gluon. On Sat, Aug 28, 2010 at 6:51 AM, Kevin extemporalgen...@gmail.com wrote: Of course, it is always an option (and never a difficulty) to wrap functional programming concepts in an object oriented wrapper -- that's more a benefit of the functional style than then anything OOP gives to you though. On Aug 27, 7:13 pm, Thadeus Burgess thade...@thadeusb.com wrote: . DON'T DO IT! The DAL is ment for functional programming... so be careful if you have to use it in a class based system. Plus you will have to have your stand alone application manage the .table files too. Just keep it in mind. In any case... You will need. gluon/LICENSE gluon/sql.py gluon/portalocker.py gluon/utils.py Also, you will NOT need validators.. so edit gluon/sql.py and comment out anything to do with the following from serializers import json - remove this, it conflicts with the real version of simplejson installed from pypi. from http import HTTP -- doubt you will be making any http exceptions import validators -- unless you are somehow using SQLFORM, you dont need this. Be sure to remove all traces of validators from the rest of sql.py Just do a search for validators and remove that code. Yay, you now have a stand alone DAL... see how easy that was to de-couple things?! =) -- Thadeus On Fri, Aug 27, 2010 at 6:34 PM, Sujan Shakya suzan.sha...@gmail.com wrote: Hi All, I'm going to have to use a MySQL orm in a project in near future. Do you think web2py DAL is good for that purpose regarding scalability and performance ? And what files do I need to import or execute to use only web2py DAL ? Thanks.
Re: [web2py] Re: Field default
On Friday 27 August 2010 8:27:31 pm mdipierro wrote:
Try
db.define_table(.,Field(...,default=Expression('CURRENT_DATE',db=db)),.
...)
I cannot promise this works but let us know.
No exceptions. What it created was:
CREATE TABLE default_test(
id INTEGER PRIMARY KEY AUTOINCREMENT,
date_default DATE,
txt_fld TEXT
);
No DEFAULT on the field.
--
Adrian Klaver
adrian.kla...@gmail.com
Re: [web2py] Re: uWSGI examples page updated for web2py
thanks. Some users have experience dropped requests with uWSGI on heavy traffic. Can you reproduce the problem? Never had specific report about this (socket misconfiguration apart), they should post relevant data to the uWSGI mailing-list. uWSGI will never take down your machine if the load goes wild, so probably they are reaching some (configurable) limit. If they experienced that on benchmarking there is a (old) procedure to follow: http://lists.unbit.it/pipermail/uwsgi/2009-December/40.html -- Roberto De Ioris http://unbit.it
