[web2py] Re: Web2py dal significantly slower than raw query

[Niphlod]

Think an int, that needs to be "casted" to an int for 50k times. You can't 
expect an abstraction layer to be as fast as raw sql. DAL doesn't just 
write queries for you, it adapts the resultset to the model. 50k rows in 
less than 2 seconds is not that bad.

On Tuesday, October 3, 2023 at 8:29:12 AM UTC+2 urban@gmail.com wrote:

>
> 12 columns, 50k rows (17k rows in the query result)
> cacheable=True is about 0.3s faster. 
>
> The costliness of building the whole model comes (mostly) from reference 
> types?
> On Monday, October 2, 2023 at 11:22:53 PM UTC+2 Niphlod wrote:
>
>> how many rows and how many columns ? the raw one is returning whatever 
>> type the underlying database structure has as it is, the pydal one is 
>> building the whole model (including references)
>> did you try with cacheable=True in the pydal one ? it won't build  
>> update_records and delete_records, for starters. 
>>
>> On Monday, October 2, 2023 at 8:04:04 PM UTC+2 Jim S wrote:
>>
>>> The only thing I can see is that the SQL needs to be 'built' by pydal, 
>>> but I find it hard to believe it takes a whole second.  Massimo might have 
>>> to add context here.
>>>
>>> -Jim
>>>
>>> On Monday, October 2, 2023 at 1:00:19 PM UTC-5 urban@gmail.com 
>>> wrote:
>>>
>>>> Sorry my bad! I mixed up the timings when editing the post.
>>>>
>>>> The slower timing is for the dal version.
>>>> Moreover the dal version is slower even if I remove the .as_list() call.
>>>> I had originally tried that. 
>>>> When I get the time I'll try "debugging" it by looking at the dal.py 
>>>> source.
>>>> Asking here if anyone knows an obvious reason for this that I am 
>>>> missing.
>>>> On Monday, October 2, 2023 at 5:53:47 PM UTC+2 Jim S wrote:
>>>>
>>>>> It's possible I'm reading this wrong (it is Monday morning), but .09s 
>>>>> (DAL) is faster than 1.8s (raw SQL).  
>>>>>
>>>>> Is that a typo?  Or, is it my Monday-morning-brain?
>>>>>
>>>>> If your raw query is slower, could it be because you're converting to 
>>>>> a dict instead of a list as in your dal query?
>>>>>
>>>>> -Jim
>>>>>
>>>>> On Monday, October 2, 2023 at 3:07:26 AM UTC-5 urban@gmail.com 
>>>>> wrote:
>>>>>
>>>>>>   ids =  tuple(m['id'] for m in relevant_models)
>>>>>>   raw_q = db.executesql("""
>>>>>>   SELECT 
>>>>>>   *
>>>>>>   FROM "table" 
>>>>>>   WHERE ("table".ref_id" IN {});
>>>>>> """.format(str(ids)), as_dict=True)
>>>>>>  1.8s
>>>>>>
>>>>>>   ids =  tuple(m['id'] for m in relevant_models)
>>>>>>   dal_q = db(
>>>>>>   db.table.ref_id.belongs(ids)
>>>>>>   ).select(db.table.ALL).as_list()
>>>>>> # 0.09s
>>>>>> *Web2Py 2.22.3*
>>>>>>
>>>>>> Why would the dal query be so much slower than the raw sql? The 
>>>>>> generated sql (db._lastsql) is the same as the raw. 
>>>>>>
>>>>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/2b768764-1060-48d0-844e-c302bff66130n%40googlegroups.com.

[web2py] Re: Web2py dal significantly slower than raw query

[Niphlod]

how many rows and how many columns ? the raw one is returning whatever type 
the underlying database structure has as it is, the pydal one is building 
the whole model (including references)
did you try with cacheable=True in the pydal one ? it won't build  
update_records and delete_records, for starters. 

On Monday, October 2, 2023 at 8:04:04 PM UTC+2 Jim S wrote:

> The only thing I can see is that the SQL needs to be 'built' by pydal, but 
> I find it hard to believe it takes a whole second.  Massimo might have to 
> add context here.
>
> -Jim
>
> On Monday, October 2, 2023 at 1:00:19 PM UTC-5 urban@gmail.com wrote:
>
>> Sorry my bad! I mixed up the timings when editing the post.
>>
>> The slower timing is for the dal version.
>> Moreover the dal version is slower even if I remove the .as_list() call.
>> I had originally tried that. 
>> When I get the time I'll try "debugging" it by looking at the dal.py 
>> source.
>> Asking here if anyone knows an obvious reason for this that I am missing.
>> On Monday, October 2, 2023 at 5:53:47 PM UTC+2 Jim S wrote:
>>
>>> It's possible I'm reading this wrong (it is Monday morning), but .09s 
>>> (DAL) is faster than 1.8s (raw SQL).  
>>>
>>> Is that a typo?  Or, is it my Monday-morning-brain?
>>>
>>> If your raw query is slower, could it be because you're converting to a 
>>> dict instead of a list as in your dal query?
>>>
>>> -Jim
>>>
>>> On Monday, October 2, 2023 at 3:07:26 AM UTC-5 urban@gmail.com 
>>> wrote:
>>>
   ids =  tuple(m['id'] for m in relevant_models)
   raw_q = db.executesql("""
   SELECT 
   *
   FROM "table" 
   WHERE ("table".ref_id" IN {});
 """.format(str(ids)), as_dict=True)
  1.8s

   ids =  tuple(m['id'] for m in relevant_models)
   dal_q = db(
   db.table.ref_id.belongs(ids)
   ).select(db.table.ALL).as_list()
 # 0.09s
 *Web2Py 2.22.3*

 Why would the dal query be so much slower than the raw sql? The 
 generated sql (db._lastsql) is the same as the raw. 



-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/21b30c28-5378-416f-86de-3e686e8023dbn%40googlegroups.com.

[web2py] Re: pyodbc error on windows machine

[Niphlod]

just as an off chance thing  what's the default language of the user ? 
never had troubles with english. Also, what's your table model definition 
and what's the DDL for the underlying table ?

On Thursday, September 7, 2023 at 10:49:29 PM UTC+2 jays...@gmail.com wrote:

> still hoping for some advice here
>
> On Thursday, September 7, 2023 at 8:32:03 AM UTC-4 jays...@gmail.com 
> wrote:
>
>> Hello all, i have an app that is connect to MSSQL and is functioning 
>> normally when run on other machine, but when i run it i'm recieving this 
>> message:
>>
>> pyodbc.DataError: ('22007', '[22007] [Microsoft][ODBC SQL Server 
>> Driver][SQL Server]Conversion failed when converting date and/or time from 
>> character string. (241) (SQLExecDirectW)')
>>
>> I'm not able to find the differences in ODBC drivers from my machine vs 
>> the machines that have no issues... but hoping someone could point me in 
>> the right direction
>>
>> Thank you!
>>
>> Jason
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/a8b13023-4870-4a67-804a-077cbc37d8e9n%40googlegroups.com.

Re: [web2py] Re: sessions folder under application

[Niphlod]

there's an handy session cleaner   web2py/sessions2trash.py at master · 
web2py/web2py (github.com) 
 and 
a whole section in the docs  web2py - Deployment recipes 


On Wednesday, July 6, 2022 at 3:22:43 PM UTC+2 Vlad wrote:

> Invalidating existing sessions is not a problem; they can login again. As 
> long as it doesn't break things... 
>
> Thank you, Jim! 
>
>
> On Wed, Jul 6, 2022, 9:20 AM Jim S  wrote:
>
>> I believe that you can.
>>
>> However, this will invalidate all active sessions and all active users 
>> will have to log in again.
>>
>> Are there date stamps you can use to delete old session files but keep 
>> current ones?
>>
>> -Jim
>>
>> On Tuesday, July 5, 2022 at 10:44:35 PM UTC-5 Vlad wrote:
>>
>>> I was checking memory usage and noticed that the sessions folder (under 
>>> one specific application) takes over 1G of disk space. Can I just delete 
>>> everything in there? Trying to clean up the space to optimize disk usage - 
>>>
>> -- 
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> --- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "web2py-users" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/web2py/VBkKItehYfQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> web2py+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/web2py/ef6ffa3f-1f27-43ed-9f70-8a8e694d501bn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/6bf23f74-2677-48bb-a1fd-6673dcd4efb7n%40googlegroups.com.

Re: [web2py] Re: How to add command line arguments to web2py when it is running as an application pool on IIS?

[Niphlod]

the timeout is part of rocket webserver, not web2py. if you're running 
fastcgi, that's where you want to set a timeout: as configuration in IIS 
(see  python - How do I stop FastCGI process exceeding the configured 
request timeout - Stack Overflow 
<https://stackoverflow.com/questions/57917122/how-do-i-stop-fastcgi-process-exceeding-the-configured-request-timeout>
  
or something like that )

On Wednesday, September 1, 2021 at 11:12:29 AM UTC+2 david@gmail.com 
wrote:

> I would like to increase the timeout parameters to see if it resolves some 
> issues we are having with internal server error messages. 
>
> Get Outlook for Android <https://aka.ms/AAb9ysg>
> --
> *From:* web...@googlegroups.com  on behalf of 
> Niphlod 
> *Sent:* Wednesday, September 1, 2021 10:35:35 AM
> *To:* web2py-users 
> *Subject:* [web2py] Re: How to add command line arguments to web2py when 
> it is running as an application pool on IIS? 
>  
> what command line argument, specifically ?
>
> On Tuesday, August 31, 2021 at 5:59:31 AM UTC+2 snide...@gmail.com wrote:
>
> On Thursday, August 26, 2021 at 11:30:34 PM UTC-7 david@gmail.com 
> wrote:
>
> Hi web2py folks, 
>
> I would like to add a command line argument to the startup of web2py which 
> is running on an IIS web server.  Can someone tell me the place where I 
> need to add those parameters on the IIS?
>
> Kind Regards,
> David
>
>
>
> I can't, but perhaps the archives can help.  One of the long-time masters, 
> Simone, ran IIS configurations and sometimes discussed his setups.  Use the 
> search string "
> https://groups.google.com/g/web2py/search?q=author%3ANiphlod; to see his 
> posts.  (There are a lot; he did the Scheduler and also the JWT 
> implementation, so you may want to refine the search.) 
>
> /dps
>
>
>  
>
> -- 
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> --- 
> You received this message because you are subscribed to the Google Groups 
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to web2py+un...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/web2py/1ac4efb0-1524-4f03-9370-4ac03da788bfn%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/web2py/1ac4efb0-1524-4f03-9370-4ac03da788bfn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/9c5ee293-acfc-42fe-aac1-6fe3f67db71fn%40googlegroups.com.

[web2py] Re: How to add command line arguments to web2py when it is running as an application pool on IIS?

[Niphlod]

what command line argument, specifically ?

On Tuesday, August 31, 2021 at 5:59:31 AM UTC+2 snide...@gmail.com wrote:

> On Thursday, August 26, 2021 at 11:30:34 PM UTC-7 david@gmail.com 
> wrote:
>
>> Hi web2py folks,
>>
>> I would like to add a command line argument to the startup of web2py 
>> which is running on an IIS web server.  Can someone tell me the place where 
>> I need to add those parameters on the IIS?
>>
>> Kind Regards,
>> David
>>
>
>
> I can't, but perhaps the archives can help.  One of the long-time masters, 
> Simone, ran IIS configurations and sometimes discussed his setups.  Use the 
> search string "
> https://groups.google.com/g/web2py/search?q=author%3ANiphlod; to see his 
> posts.  (There are a lot; he did the Scheduler and also the JWT 
> implementation, so you may want to refine the search.)
>
> /dps
>
>
>  
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/1ac4efb0-1524-4f03-9370-4ac03da788bfn%40googlegroups.com.

[web2py] Re: nssm does not work

[Niphlod]

what happened may differ from what you thought it should happen which may 
differ from what needs to happen.

said that, nssm just bootstrap the process. the fact that you have a 
service up may already indicate that everything is running smoothly 
but if you wanna do a check, open the task manager, go to details, see if 
there's a web2py running (it may be useful to select the column holding the 
command line)

On Monday, June 3, 2019 at 8:58:02 AM UTC+2, rāma wrote:
>
> I am using the latest version of NSSM - the Non-Sucking Service Manager; 
> both 2.24 and the prelease build 2.2.4-101 .
>
> Unfortunately for me, NSSM sucked big time and trying to get it working 
> with web2py was an utter waste of time.
>
> Tried
> -K; nothing and tried
> -X; nothing anyways
>
> Env:
> Python3.7
> Windows 10
>
> Can anyone advise what is supposed to happen after issuing -K app1 ? A 
> windows pop-up or the local webserver starts ? There is no outcome clearly 
> explained in any material I have gone through so far.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/web2py/7625baa6-0e7f-4bcc-973a-c3e79dfc5e43%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Scheduler tasks fail intermitently with no apparent reason (sys.exit(1))

[Niphlod]

Timeout?!

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler Replacement

[Niphlod]

jokes aside, yeah, you definitely hit scheduler's limits, or, for better 
saying, limits using a relational database as a queue table.
web2py's scheduler can still be optimized, and I feel that 30k tasks are 
manageable, if they are spread throughout the day (20 tasks a minute if 
math is not failing me)
Managing 15-20 workers is not a problem with a good database backend, 30ish 
or more is asking for disasters to happen, and that's why redis_scheduler 
was born.
The redis backed one moves to redis the heavy concurrency part, which is 
the scheduler_worker table and the polling algo for looking for new tasks, 
but it doesn't move EVERYTHING out of the database.
60-80 workers is probably the limit for redis_backed.
Both of them are prolly less than 2k lines of code and they use DAL and 
redis, but nothing else from the standard library so they're definitely 
good, but not on par with "specialized" alternatives.

One "grin" of the implementation is that it forks a new process every task, 
and that's for a basic design principle others do not enforce which is to 
terminate a long running process going into timeout (and that's because 
python can't kill a thread, just a process). But that's a problem for 
people running 40 tasks per second, not 20 a minute.

Things to look out for when you reach high numbers like you (i.e. you can 
see if it helps "staying with the scheduler"):
- trim the scheduler_task regularly (having 1m rows in the scheduler_task 
table, if you do not need them for reporting, can definitely bring down 
performances)
- same deal for scheduler_run (do you need the result, and if you need it, 
do you need for how much time?!)
- create indexes on it-s- the scheduler_task table (one on status, 
assigned_worker_name helps the workers,  on stop_time, next_run_time and 
status helps the ticker)

took that out of the table, you can quite easily jump on a different ship 
(my advice would be to look at rq before celery, but YMMV) which are HUGE 
projects.

The only problem you may have is that if you use the same backend to store 
results for your tasks (because they're needed)  you *may* hit the same 
limits, i.e. you may have your backend not sized up for the work your apps 
need to do.


-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler Replacement

[Niphlod]

https://gph.is/1eNXnLk 



On Wednesday, November 21, 2018 at 10:19:40 PM UTC+1, Dave S wrote:
>
>
>
> On Wednesday, November 21, 2018 at 10:31:13 AM UTC-8, Boris Aramis Aguilar 
> Rodríguez wrote:
>>
>> TL;DR : Do you guys know or has experience with an alternative scheduler 
>> to web2py's one that you would recommend? we use features such as 
>> scheduling tasks at a specific time, repetitions, timeouts, and check the 
>> task current state. Any further comments, advice and experience is really 
>> appreciated.
>> [...]
>> Do anyone of you guys know or has experience with an alternative 
>> scheduler? I've seen several options (rabbit-mq, python-rq, mrq, so on); 
>> but I'm not sure about the limitations of those schedulers... any further 
>> comments are really appreciated.
>>
>
> Nope, and my scheduler runs all of 8 tasks a day (creeping up to 16), so 
> I've hardly pushed the limit.
>
> But I would look at just running web2py instances on those other servers, 
> and having the "master" do nothing more than send https requests to each of 
> them, and each local scheduler then deals with its own workers only.  Not 
> quite like sharding on Elastisearch, but definitely division of labor.
>
> If Niphlod was still monitoring the group, that would get you the best 
> comments on the in-box scheduler.  He's the expert -- wrote the current 
> scheduler and the test suite, runs databases on diverse systems (including 
> MSSQL behind IIS, IIRC). He also did the JWT implementation.
>
> /dps
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler: Is it possible to assign a task to all workers that are part of a group?

[Niphlod]

hint: if the api doesn't allow it, it's not an envisioned scenario.

That being said, there's no limit on the worker-group configuration. 
As auth relies on RBAC, workers rely on groups. 
A particular group is able to process a given task ? Queue the task 
assigning the correct group.
Only one worker is able to process tasks of a specific group ? voilà!
It's perfectly fine to run 10 workers and have 10 groups, even if each 
group holds just one worker...

On Monday, May 7, 2018 at 8:13:02 PM UTC+2, Francisco Ribeiro wrote:
>
> I have been testing my own hypothesis, it didn't work. I was unable to 
> assign a task to a specific worker, I can only assign tasks to groups. 
> Please correct me if I am wrong...
>
> On Sunday, 6 May 2018 16:16:08 UTC+1, Francisco Ribeiro wrote:
>>
>> Thank you for confirming.
>>
>> I presume that I still can take a list of workers belonging to a single 
>> group and assign the same task multiple times (once per worker), which 
>> would still suit my needs (expect if i want to cancel it for whatever 
>> reason which becomes harder / less convenient). Do you see any problem 
>> doing that?
>>
>> On Sunday, 6 May 2018 04:16:53 UTC+1, Massimo Di Pierro wrote:
>>>
>>> No. I do not believe there is any way to guarantee that every work will 
>>> execute the same task.
>>>
>>>
>>> On Friday, 4 May 2018 17:36:30 UTC-5, Francisco Ribeiro wrote:

 The question refers to submitting a task simultaneously to all workers 
 within a group.

 Thank you

>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: How to force browser to cache css and js ?

[Niphlod]

we have a section on the book on static asset management

http://web2py.com/books/default/chapter/29/04/the-core#Static-asset-management

On Sunday, June 25, 2017 at 10:01:27 PM UTC+2, Ramos wrote:
>
> I´m using this 
> https://github.com/mjbeller/web2py-starter
> but i noticed that with only me as the single user so far it occupies 
> daily from 200Mb to 300Mb bandwidth 
>
> How can i tweek it to lower this value before going to production and 
> explode my paid webfaction hosting ?
>
> Regards
> António
>
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Suddenly my task scheduler stopped working

[Niphlod]

it's normal if the execution timeoutted.

On Tuesday, June 27, 2017 at 7:11:12 PM UTC+2, Ramos wrote:
>
> this is the error i see in task_run table record with status failed
>
> Traceback (most recent call last):
>   File "/.../new2py/web2py/gluon/scheduler.py", line 293, in executor
> _env = env(a=a, c=c, import_models=True)
>   File "/home/ramstein74/webapps/new2py/web2py/gluon/shell.py", line 166, 
> in env
> sys.exit(1)
> SystemExit: 1
>
>
> What can this be ?
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: queue_task :: trouble with quoted json strings for pvars and args

[Niphlod]

seems more a problem of the adapter rather than the scheduler itself

On Thursday, June 8, 2017 at 5:16:32 AM UTC+2, Richard P wrote:
>
> I just came across this same issue when using the web2py scheduler for the 
> first time with an Oracle DB. I spent a good amount of time googling 
>  before finding this post which pointed me to where the problem was.
>
> I have implemented a work around  by creating the below trigger in the 
> database that will strip out the single quotes before the data is inserted 
> into the table. 
> It's not pretty but it works. Now my tasks are running fine.
>
> create or replace trigger replace_quotes
> before insert on SCHEDULER_TASK
> for each row
> begin
> :new.args := replace( :new.args, , '' );
> :new.vars := replace( :new.vars, , '' );
> end;
> /
>
>
>
> On Thursday, 8 September 2016 14:27:15 UTC+1, Erwn Ltmann wrote:
>>
>> Hi Dave,
>>
>> thx for your response. I'm sorry, but it doesn't helps me. My problem 
>> concerning the usage of scheduler_task queueing. The web2py scheduler 
>> transforms automatically the dictionary parameter 'pvars' into a string 
>> represented JSON in order to save this value as VARCHAR2(512) into the vars 
>> column of database (oracle) table scheduler_task. So far so good. But IMHO 
>> with unnecessary quotes at begin and end. The scheduler part of web2py 
>> tried to read scheduler records and fails because this (damn) extra quotes.
>>
>> E.
>>
>> On Thursday, September 8, 2016 at 4:01:37 AM UTC+2, Dave S wrote:
>>>
>>>
>>>
>>> On Wednesday, September 7, 2016 at 5:30:49 PM UTC-7, Erwn Ltmann wrote:

 Hi, 

 I'm used to create a new task for example: 

 scheduler.queue_task( 
   "a_task", 
   pvars = dict(csid=row.id, csname=row.name, action=row.action), 
   ... 
 ) 

 In the past (version 2.10.1), everything was fine. Within the current 
 version (2.14.6) I have some trouble. See this code for example: 

 for row in db(db.scheduler_task.group_name == "agents").select( 
 db.scheduler_task.ALL): 
  ... 
  pvars = json.loads(row.vars) 

 ... will raise this exception: 

   ... 
   File "/usr/local/lib/python2.7/json/decoder.py", line 384, in 
 raw_decode 
 raise ValueError("No JSON object could be decoded") 
 ValueError: No JSON object could be decoded 

 The reason is a quoted string in column 'VARS' in table 
 'SCHEDULER_TASK' (no quotes in web2py's old version 2.10.1). 

 A new value for 'VARS' with quotes for example: 

 '{"action": "script", "csname": "list", "csid": 121}' 

 I can skip the quotes with a trick: 

 pvars = json.loads(row.vars[1:-1]) 

 But, it is not a general solution. The same problem appears for column 
 'ARGS' in scheduler.py when I process the tasks: 

   File "/export/home/armadm/dev01/release/web2py/gluon/scheduler.py", 
 line 311, in executor 
 args = _decode_list(loads(task.args)) 
   File "/usr/local/lib/python2.7/json/__init__.py", line 338, in loads 
 return _default_decoder.decode(s) 
   File "/usr/local/lib/python2.7/json/decoder.py", line 366, in decode 
 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 
   File "/usr/local/lib/python2.7/json/decoder.py", line 384, in 
 raw_decode 
 raise ValueError("No JSON object could be decoded") 
 ValueError: No JSON object could be decoded 

 I use Oracle as database: 

 SQL> desc scheduler_task 
  Name  Null?Type 
  -  
  
  IDNOT NULL NUMBER 
  ... 
  ARGS  VARCHAR2(512) 
  VARS  VARCHAR2(512) 

 Any ideas? 

 Thx, Erwn 

>>>
>>> the vars field is a dict, with the var names as keys and the stuff you 
>>> want to pass as the values.  Both are strings.
>>>
>>> I use a single var in my code, the name is "where" and the value is a 
>>> path.  So I do
>>> if vars["where"]:
>>>mywhere = vars["where"]
>>>
>>>
>>> No stripping of quotes is needed to get the path.
>>>
>>> /dps
>>>
>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Should I modify response.headers in order to get nginx's uwsgi_cache work properly?

[Niphlod]

It'd be easier to see if it works with a slightly modified directive for 
uwsgi_cache_valid

uwsgi_cache_valid any  1m


Because if you miss the 3 parameter notation, only 200 responses are cached.

Then you can try returning a proper X-Accel-Expires that should "trump" any 
other header.

uwsgi_cache is an application cache (read: you could use it instead of 
redis if your whole app is deployed through that nginx process)

If you're not looking for an IPC cache that is uwsgi_cache (that's 
basically what cache.ram does for single processes), maybe it'd be better 
to use the upstream proxy cache via proxy_cache and proxy_cache_path 
directive.

BTW:  $upstream_cache_status that you use shouldn't be used to see if 
uwsgi_cache is used at all. It's the "flag" that the upstream cache is 
working (proxy_cache directive) 

On Monday, April 17, 2017 at 2:15:21 PM UTC+2, Lisandro wrote:
>
> I've been dealing with this problem for some time now.
> Some weeks ago I posted a question in stackoverflow, but I didn't find a 
> solution yet:
>
> http://stackoverflow.com/questions/43052276/nginx-why-isnt-uwsgi-cache-working-with-this-headers
>
> Also I've been working with a sysop who knows better than me, but still 
> couldn't solve the problem. 
>
> I don't think web2py has anything to do with it, so I'll mark this thread 
> as "no action required".
> Still, any comment or suggestion will be appreciated.
>
> Best regards,
> Lisandro
>
> El viernes, 24 de marzo de 2017, 10:00:06 (UTC-3), Lisandro escribió:
>>
>> I'm running a web2py website with public articles, and there are 
>> ocasional peaks in traffic. 
>> I use Nginx as a webserver, and uWsgi to run my web2py application.
>>
>> Considering the articles are public (the HTML page of an article is the 
>> same for every visitor), I'm already doing some caching in order to improve 
>> performance (I'm using @cache.action decorator with Redis model).
>> However, and please correct me if I'm wrong, for every request made to 
>> the URL of an article, before the caching can be done, the models need to 
>> be executed.
>> So I thought I could improve performance even more, caching the HTML 
>> directly from Nginx, that way I would save resources in my server.
>>
>> However I'm having a hard time getting it, and I wanted to know if I 
>> should modify the response.headers.
>> I've read that they come set by default:
>> http://web2py.com/books/default/chapter/29/04/the-core#response
>>
>> To do some tests, I have this simple web2py function:
>>
>> def test():
>> from datetime import datetime
>>
>> return datetime.now().strftime('%H:%M:%S')
>>
>>
>>
>> In the ngin'x side, the server block configuration is this:
>>
>> uwsgi_cache_path /tmp/nginx_cache/ levels=1:2 keys_zone=mycache:10m 
>> max_size=10g inactive=10m use_temp_path=off;
>>
>> server {
>> ...
>>
>> location / {
>> # response header to check if cache is a HIT or a MISS
>> add_header  X-uWSGI-Cache $upstream_cache_status;
>>
>> # server cache
>> uwsgi_cache  mycache;
>> uwsgi_cache_valid  15m;
>> uwsgi_cache_key  $request_uri;
>>
>> # client cache
>> expires 3m;
>>
>> uwsgi_pass  unix:///tmp/web2py.socket;
>> include uwsgi_params;
>> uwsgi_param UWSGI_SCHEME $scheme;
>> }
>> }
>>
>>
>> But every time I hit the test page, I check the response headers and I 
>> see always a MISS.
>> In other words, nginx still sends the requests to uwsgi, and the page is 
>> generated in every request.
>> I've found this forum post where someone says this:
>>
>> *"...it looks to me like the issue is that the upstream server is just 
>> not sending response that contain an expiration date (Expires:) or a cache 
>> validator (for instance, Last-Modified:). (The cookie expiration time has 
>> nothing to do with caching.)*
>> *The HTTP 1.1 spec 
>>  says: 'If 
>> there is neither a cache validator nor an explicit expiration time 
>> associated with a response, we do not expect it to be cached, but certain 
>> caches MAY violate this expectation (for example, when little or no network 
>> connectivity is available).'"*
>>
>>
>> So I thought I would still needed to use the @cache.action decorator 
>> (with model=None in order to only set response headers to allow client 
>> caching):
>>
>> @cache.action(time_expire=222, cache_model=None, session=False, vars=
>> False, public=True)
>> def test():
>> from datetime import datetime
>>
>> return datetime.now().strftime('%H:%M:%S')
>>
>>
>> However I sill can't get it to work.
>> I set up time_expire=222 to check if the directive "expires 3m;" in 
>> nginx's configuration would overwrite it, and yes it does, the responses 
>> have a Cache-Control: max-age=180 (that is 3 minutes, not 222 seconds).
>>
>> *I don't intend to talk about nginx's configuration variables, but I'm 
>> tempted to 

[web2py] Re: ProgrammingError: column "worker_stats__tmp" is of type json but expression is of type text

[Niphlod]

you didn't let migration happen after dropping both the .table files and 
the underlying tables on the database.

On Wednesday, March 22, 2017 at 6:20:12 PM UTC+1, Ramos wrote:
>
> I had the same problem this week and did all 
>
>
> *Please drop the scheduler_worker table from the database and remove the 
> file *_scheduler_worker.table from the databases/ folder of your 
> application, then set migrate to True (a single request with migrate=True 
> will suffice). The table will be recreated properly and the error will go 
> away.*
> Now i get the error
>  relation "scheduler_worker" does not 
> exist
>
> any help?
>
>
> Em quarta-feira, 8 de março de 2017 14:05:46 UTC, Ben Lawrence escreveu:
>>
>> forget it, I am going back to mysql. 
>>
>> On Sunday, March 5, 2017 at 4:48:30 AM UTC-8, Ben Lawrence wrote:
>>>
>>> OK,
>>> I have two machines, one with web2py+nginx (machine A) and the other 
>>> with postgresql (machine B)
>>>
>>> On machine A:
>>> I check migrate = 0
>>> I delete all the scheduler tables
>>> I check that I deleted all the scheduler tables
>>>
>>> On machine B:
>>> I delete all the scheduler tables in postgres
>>>
>>> On machine A:
>>> I check that there are no scheduler tables.
>>> I reboot it
>>>
>>> I check that there are no scheduler tables
>>> I turn on migrate = 1
>>> There are now scheduler tables.
>>> I go to scheduler tables in web2py admin, scheduler_tasks show new tasks 
>>> that have since completed,
>>> scheduler_run show new tasks that have completed,
>>> but scheduler_worker has error:
>>>
>>> Traceback (most recent call last):
>>>   File 
>>> "/home/www-data/web2py/applications/temperature/controllers/appadmin.py:select",
>>>  line 270, in select
>>>   File "/home/www-data/web2py/gluon/packages/dal/pydal/objects.py", line 
>>> 2045, in select
>>> return adapter.select(self.query, fields, attributes)
>>>   File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", 
>>> line 746, in select
>>> return self._select_aux(sql, fields, attributes, colnames)
>>>   File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", 
>>> line 727, in _select_aux
>>> return processor(rows, fields, colnames, cacheable=cacheable)
>>>   File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", 
>>> line 305, in parse
>>> for row in rows
>>>   File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", 
>>> line 229, in _parse
>>> value = self.parse_value(value, fit, ft, blob_decode)
>>>   File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", 
>>> line 196, in parse_value
>>> return self.parser.parse(value, field_itype, field_type)
>>>   File 
>>> "/home/www-data/web2py/gluon/packages/dal/pydal/parsers/__init__.py", line 
>>> 101, in parse
>>> return self.registered[field_itype](value, field_type)
>>>   File 
>>> "/home/www-data/web2py/gluon/packages/dal/pydal/parsers/__init__.py", line 
>>> 76, in __call__
>>> return self.call(value, field_type)
>>>   File 
>>> "/home/www-data/web2py/gluon/packages/dal/pydal/parsers/__init__.py", line 
>>> 73, in _call
>>> return self.f(self.parser, value)
>>>   File "/home/www-data/web2py/gluon/packages/dal/pydal/parsers/base.py", 
>>> line 129, in _json
>>> raise RuntimeError('json data not a string')
>>> RuntimeError: json data not a string
>>>
>>>
>>> On Friday, August 26, 2016 at 4:41:38 PM UTC-7, Ben Lawrence wrote:
>>>>
>>>> Thanks, I must have got the sequence wrong.
>>>>
>>>> On Thursday, August 25, 2016 at 12:05:38 PM UTC-7, Niphlod wrote:
>>>>>
>>>>> if you get any error with __tmp is because you didn't drop tables on 
>>>>> the backend AND .table files before hitting the app with migrate=True.
>>>>>
>>>>> On Thursday, August 25, 2016 at 6:53:07 PM UTC+2, Ben Lawrence wrote:
>>>>>>
>>>>>> Hi Niphlod,
>>>>>> I deleted all the scheduler tables , created a new database and then 
>>>>>> rebooted with DAL(..migrate=True..) yet still get this error
>>>>>> column "

[web2py] Re: info: interesting hack, sort in IS_IN_DB lists with regard to current locale

[Niphlod]

BTW: setlocale is not threadsafe.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler is_ticker and deadlock issues

[Niphlod]

it really doesn't matter. the IPC is done on the database, so having local 
workers hitting it or remote ones doesn't turn into more transactions.

On Friday, January 27, 2017 at 6:08:56 PM UTC+1, Jason Solack wrote:
>
> In your scenario do you have 10 workers on multiple machines?   So having 
> 5 workers on 1 machine and 5 on another?  we can easily do 10 workers, but 
> we run into issues when we have them split across machines
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: SystemExit: 1 ticket on scheduler

[Niphlod]

or it got a SIGTERM, or it was timing out. 

On Monday, January 30, 2017 at 10:16:49 PM UTC+1, Anthony wrote:
>
> Looks like maybe an error somewhere in your model files. Is that the 
> traceback from the associated record in the scheduler_run table?
>
> Anthony
>
> On Monday, January 30, 2017 at 8:02:29 AM UTC-5, Gael Princivalle wrote:
>>
>> Hello.
>>
>> I've got a scheduler function that run each 30 seconds for sending web 
>> push notifications:
>>
>> def process_webn_alerts():
>> webn_alerts = db(db.alerts.instant_webn_status == 1).select()
>> for webn_alert in webn_alerts:
>> onesignal_players = db(db.onesignal_players.created_by == 
>> webn_alert.to_user).select()
>> for onesignal_player in onesignal_players:
>> header = {"Content-Type": "application/json; charset=utf-8",
>> "Authorization": "Basic my_auth_key"}
>> payload = {"app_id": "my_app_id",
>> "headings": {"en": webn_alert.title},
>> "include_player_ids": [onesignal_player.uuid],
>> "contents": {"en": webn_alert.body},
>> "url": webn_alert.link,
>> "chrome_web_icon": webn_alert.icon_url
>> }
>> req = requests.post("
>> https://onesignal.com/api/v1/notifications;, headers=header, data=json.
>> dumps(payload))
>> print(req.status_code, req.reason)
>> db(db.alerts.id == webn_alert.id).update(instant_webn_status 
>> = 2)
>> db.commit()
>>
>> It has run correctly for 10 hours, without any web push notification to 
>> send. After that it failed with this ticket:
>>
>> Traceback (most recent call last):
>>   File "/home/tasko/webapps/w2p_2_14_16/web2py/gluon/scheduler.py", line 
>> 293, in executor
>> _env = env(a=a, c=c, import_models=True)
>>   File "/home/tasko/webapps/w2p_2_14_16/web2py/gluon/shell.py", line 166, 
>> in env
>> sys.exit(1)
>> SystemExit: 1
>>
>> Does it mean that the http Onesignal call has failed?
>>
>> Thanks.
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler is_ticker and deadlock issues

[Niphlod]

I think I posted the relevant number of queries issued to the backend for a 
given number of workers but I do daily use the scheduler on an mssql db and 
it can easily handle at least 10 workers (with the default heartbeat). 
Locking kicks in maybe once or twice a day, which means 1 or 2 on 28800 
occasions, which is a pretty damn low number :P
Of course the backend *should* be able to sustain concurrency, but on a 
minimal server with very low specs 6 or 7 workers should absolutely pose no 
threats at all. 
For 5 workers all that is needed is a backend being able to handle 240 
transactions per minute!

On Thursday, January 26, 2017 at 7:47:51 PM UTC+1, Dave S wrote:
>
> On Thursday, January 26, 2017 at 9:45:20 AM UTC-8, Jason Solack wrote:
>>
>> using mssql, the code itself is in gluon scheduler.py - this happens with 
>> no interaction from the app
>>
>>
> How do you instantiate the Scheduler?
>
> Is the mssql engine on the same machine as any of the web2py nodes?  Are 
> there non-web2py connections to it?
>
> /dps
>  
>
>> On Thursday, January 26, 2017 at 12:03:41 PM UTC-5, Dave S wrote:
>>>
>>>
>>>
>>> On Thursday, January 26, 2017 at 8:44:25 AM UTC-8, Jason Solack wrote:
>>>>
>>>> So the issue is we run 6 workers on a machine and it works.  If we do 3 
>>>> workers on 2 machines we get deadlocks.  That is no exaggeration - 6 
>>>> records in our worker table and we're getting dealocks.
>>>>
>>>>
>>> Which DB are you using?  Can you show your relevant code?
>>>
>>> /dps
>>>  
>>>
>>>> On Wednesday, January 25, 2017 at 3:05:37 AM UTC-5, Niphlod wrote:
>>>>>
>>>>> you *should* have one different db for each environment. Each 
>>>>> scheduler tied to the same db will process incoming tasks, and it doesn't 
>>>>> matter what app effectively pushes them.
>>>>> This is good if you want to have a single scheduler (which can be 
>>>>> composed by several workers) serving many apps, but *generally* you don't 
>>>>> want to *merge* prod and beta apps.
>>>>>
>>>>> The is_ticker bit is fine: only one worker tied to a db is elegible to 
>>>>> be a ticker, which is the one process than manages asssigning tasks (to 
>>>>> itself AND to other available workers).
>>>>> Locking, once in a while, can happen and is self-healed. Continuous 
>>>>> locking is not good: either you have too many workers tied to the db OR 
>>>>> your db isn't processing concurrency at the rate that it needs. 
>>>>> SQLite can handle at most 2 or 3 workers. All the other "solid" 
>>>>> backends can manage up to 10, 15 at most.
>>>>> If you wanna go higher, you need to turn to the redis-backed scheduler.
>>>>>
>>>>> On Tuesday, January 24, 2017 at 10:59:31 PM UTC+1, Jason Solack wrote:
>>>>>>
>>>>>> Hello all, 
>>>>>>
>>>>>> I'm having some re-occurring issue with the scheduler.  We are 
>>>>>> currently running multiple environments (production, beta) and have 
>>>>>> several 
>>>>>> nodes in each environment.  If we have scheduler services running on all 
>>>>>> machines on each node we get a lot of deadlock errors.  If we drop each 
>>>>>> environment down to one node we get no deadlock errors.  I am noticing 
>>>>>> the 
>>>>>> field "is_ticker" in the worker table will only have one ticker across 
>>>>>> all 
>>>>>> the workers (spanning environments).  Is that the expected behavior?  I 
>>>>>> don't see any documentation about the ticker field so i'm not sure what 
>>>>>> to 
>>>>>> expect from that.
>>>>>>
>>>>>> Also is there any best practices about running the scheduler in an 
>>>>>> environment that i've described?  
>>>>>>
>>>>>> Thanks in advance
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler is_ticker and deadlock issues

[Niphlod]

you *should* have one different db for each environment. Each scheduler 
tied to the same db will process incoming tasks, and it doesn't matter what 
app effectively pushes them.
This is good if you want to have a single scheduler (which can be composed 
by several workers) serving many apps, but *generally* you don't want to 
*merge* prod and beta apps.

The is_ticker bit is fine: only one worker tied to a db is elegible to be a 
ticker, which is the one process than manages asssigning tasks (to itself 
AND to other available workers).
Locking, once in a while, can happen and is self-healed. Continuous locking 
is not good: either you have too many workers tied to the db OR your db 
isn't processing concurrency at the rate that it needs. 
SQLite can handle at most 2 or 3 workers. All the other "solid" backends 
can manage up to 10, 15 at most.
If you wanna go higher, you need to turn to the redis-backed scheduler.

On Tuesday, January 24, 2017 at 10:59:31 PM UTC+1, Jason Solack wrote:
>
> Hello all, 
>
> I'm having some re-occurring issue with the scheduler.  We are currently 
> running multiple environments (production, beta) and have several nodes in 
> each environment.  If we have scheduler services running on all machines on 
> each node we get a lot of deadlock errors.  If we drop each environment 
> down to one node we get no deadlock errors.  I am noticing the field 
> "is_ticker" in the worker table will only have one ticker across all the 
> workers (spanning environments).  Is that the expected behavior?  I don't 
> see any documentation about the ticker field so i'm not sure what to expect 
> from that.
>
> Also is there any best practices about running the scheduler in an 
> environment that i've described?  
>
> Thanks in advance
>
> Jason
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Are executesql parameters escaped?

[Niphlod]

when you do that, you're passing 'string'.format() (which is a string) to 
executesql.

the proper way is to use parameters!!!

db.executesql('select * from auth_user where id = ?', placeholders=(1, ), 
as_dict=True)

the nitty gritty details of the syntax depend on the driver parameter

On Thursday, January 12, 2017 at 2:46:28 PM UTC+1, Jorrit wrote:
>
> When I use *db.executesql* like so:
>
> birthdays = db.executesql("SELECT * FROM auth_user WHERE  DAYOFYEAR(curdate() 
> -2) < dayofyear(dateOfBirth) "
>   "AND DAYOFYEAR(curdate()) +7 >= 
> dayofyear(dateOfBirth) "
>   "AND employeeState_id = {1} AND location_id IN 
> (SELECT location_id FROM clusterLocation WHERE cluster_id={0}) ORDER BY 
> MONTH(dateOfBirth), "
>   "DAY(dateOfBirth);".format(cluster, active_id), 
> as_dict=True)
>
>
> ...are the cluster and active_id parameters SQL-escaped? If not what is 
> the best way to do this?
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Selecting booleans from sqlite3 database

[Niphlod]

did you try dropping the databae and letting web2py recreate it ? sqlite 
doesn't go well with fields altering their type "afterwards" (i.e. maybe 
you created it as a string and then turned into a boolean)

On Monday, January 9, 2017 at 9:17:19 PM UTC+1, LoveWeb2py wrote:
>
> Hello,
>
>
> I'm trying to do a simple query against a sqlite3 database where the 
> values are stored as 0's and 1's
>
> All i want to do is select all values that are True or False
>
> According to the book this should work:
>
> db(db.table.field==True).select()
>
> and that would return rows
>
> however, it's returning Rows(0) (nothing)
>
> I verified in the database that they are being stored as True/False 
> through appadmin
>
> Any thoughts on why this isn't working?
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Niphlod]

never though about just scheduling the functions via the scheduler ?

On Thursday, January 5, 2017 at 6:54:59 AM UTC+1, Brendan Barnwell wrote:
>
> I have a situation where I have (or am going to have) multiple 
> applications running on a single web2py instance.  Each application will do 
> its own thing, but they are all using the same databases.  Some of the 
> applications may serve as backend API endpoints where certain kinds of data 
> can be retrieved.  Other applications may sometimes want to make calls to 
> these endpoints to get the data and use it in their own operations.
>
> So what I'm wondering is: is there any way for one application to call a 
> controller in another application and get the raw data that WOULD be used 
> to build an actual HTTP response, but without actually going through the 
> network?  That is, suppose I have and endpoint /app1/controller1/get_data 
> and another /app2/controller2/do_something .  Inside do_something, I want 
> to get the data that get_data gets.
>
> Currently I have get_data set up as a JSON service.  I could certainly 
> make the request to the other endpoint, but that involves a number of extra 
> steps: generating an outgoing HTTP request, serializing any data to passed 
> with it, generating the incoming HTTP request object, deserializing the 
> request data, then serializing the response and deserializing it again when 
> it's received.  What I really want to do is to just call the function 
> get_data AS IF a request had been submitted, but have everything happen on 
> the server, without any network requests actually happening.  As far as I 
> can tell, I can't in any way "import" the get_data function from the other 
> app, because if I call it, it won't have the magic environment set up 
> (e.g., "db" won't exist for it to do its database queries).  I would even 
> be okay with "pretending" like I was doing a request, as long as web2py 
> could somehow realize that the app I want to call is running on the same 
> web2py instance, and call it "directly" instead of submitting a network 
> request to itself.
>
> I realize that this architecture involves apps that are more 
> interdependent than they would be in a "by the book" web2py project.  But 
> basically the idea is that the functionality of the apps is independent, 
> and in theory they should work if running on different servers, by making 
> real requests.  It's just that, IF they are running on the same server, I'd 
> like to communicate between them without going through the network, to 
> improve performance.  Also, I can't really change the architecture since 
> it's for a project I'm working on but don't ultimately control.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Simple scheduler issue

[Niphlod]

you're probably referring to -X .

in any case, web2py.py -h is your best friend ;-)

On Tuesday, January 3, 2017 at 12:47:39 AM UTC+1, Aydin wrote:
>
> I cann't believe I have spent 3 days trying every thing to find out what 
> the hell is happening. Last time I used it, it was 2 years ago and I guess 
> it was more intuitive to feel like python web2py.py -a pass -K myapp will 
> run both the server and scheduler!!! Sorry everyone for the confusion.
>
> On Monday, January 2, 2017 at 6:04:57 PM UTC-5, Niphlod wrote:
>>
>> if you run a single process with 
>>
>> python web2py.py -K appname 
>>
>> ONLY the scheduler will be alive
>>
>> the web-serving part needs to be started as usual, in addition to the 
>> previous process which just spins the scheduler process.
>>
>> On Monday, January 2, 2017 at 8:28:37 PM UTC+1, Aydin wrote:
>>>
>>> I had perfectly fine working app using scheduler in 2015, I tried to use 
>>> that one but when trying to run the schedule (-K my app), the app does not 
>>> run and web2py gets stuck in "starting single-scheduler...". This says to 
>>> me something in terms of python and python tools version does not work well 
>>> with web2py. I don't think there is anything wrong with the code I am using.
>>>
>>> On Monday, January 2, 2017 at 1:58:25 PM UTC-5, Aydin wrote:
>>>>
>>>> Sure, here are the databases I have tried to create:
>>>>
>>>> # db2 = DAL('sqlite://scheduler_storage.sqlite')
>>>> # db2 = DAL('pymysql://root:pass@localhost/testdb2')
>>>> db2=DAL('mysql://root:pass@localhost/testdb2')
>>>>
>>>> # db2 = DAL('MySQLdb://root:pass@localhost/testdb2')
>>>> # db2 = DAL('pymysql://root:pass@localhost/testdb1')
>>>>
>>>> for MySQLdb and pymysql I get the error that they are not supported. 
>>>> mysql does not give that error but gets stuck in starting the task.
>>>> I even tried to import pymysql in web2py.py which did not help with the 
>>>> error of it being not supported.
>>>>
>>>> On Monday, January 2, 2017 at 1:38:42 PM UTC-5, Dave S wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Monday, January 2, 2017 at 8:22:55 AM UTC-8, Aydin wrote:
>>>>>>
>>>>>> I used the exact example shown by massimo (https://vimeo.com/27478796) 
>>>>>> and it gets stuck at (starting single-scheduler for myapp...) and the 
>>>>>> website does not come up saying browser cannot establish a connection). 
>>>>>> Running on ubuntu and debian.
>>>>>>
>>>>>
>>>>> Can you show us your connection string (passwords obfuscated)?
>>>>>
>>>>> /dps
>>>>>  
>>>>>
>>>>>>
>>>>>> On Monday, January 2, 2017 at 11:04:44 AM UTC-5, Niphlod wrote:
>>>>>>>
>>>>>>> scheduler definitely works for mysql. 
>>>>>>> what you experienced is perfectly fine. a scheduler process just 
>>>>>>> sits there if there is nothing to do (i.e. no tasks to process).
>>>>>>>
>>>>>>> On Sunday, January 1, 2017 at 5:14:21 PM UTC+1, Aydin wrote:
>>>>>>>>
>>>>>>>> I have used the scheduler before without being worried about the 
>>>>>>>> database type. 
>>>>>>>> I tried to use mysql but it gave an error that it is not supported. 
>>>>>>>> I have mysql and pymysql installed and created a database "testdb" and 
>>>>>>>> used 
>>>>>>>> the manual format and got that error that it is not supported.
>>>>>>>>
>>>>>>>> On Saturday, December 31, 2016 at 4:09:23 PM UTC-5, 黄祥 wrote:
>>>>>>>>>
>>>>>>>>> perhaps, it related with database issue such as : sqlite
>>>>>>>>> had you already tried another database like mysql or postgresql?
>>>>>>>>>
>>>>>>>>> best regards,
>>>>>>>>> stifan
>>>>>>>>>
>>>>>>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Simple scheduler issue

[Niphlod]

if you run a single process with 

python web2py.py -K appname 

ONLY the scheduler will be alive

the web-serving part needs to be started as usual, in addition to the 
previous process which just spins the scheduler process.

On Monday, January 2, 2017 at 8:28:37 PM UTC+1, Aydin wrote:
>
> I had perfectly fine working app using scheduler in 2015, I tried to use 
> that one but when trying to run the schedule (-K my app), the app does not 
> run and web2py gets stuck in "starting single-scheduler...". This says to 
> me something in terms of python and python tools version does not work well 
> with web2py. I don't think there is anything wrong with the code I am using.
>
> On Monday, January 2, 2017 at 1:58:25 PM UTC-5, Aydin wrote:
>>
>> Sure, here are the databases I have tried to create:
>>
>> # db2 = DAL('sqlite://scheduler_storage.sqlite')
>> # db2 = DAL('pymysql://root:pass@localhost/testdb2')
>> db2=DAL('mysql://root:pass@localhost/testdb2')
>>
>> # db2 = DAL('MySQLdb://root:pass@localhost/testdb2')
>> # db2 = DAL('pymysql://root:pass@localhost/testdb1')
>>
>> for MySQLdb and pymysql I get the error that they are not supported. 
>> mysql does not give that error but gets stuck in starting the task.
>> I even tried to import pymysql in web2py.py which did not help with the 
>> error of it being not supported.
>>
>> On Monday, January 2, 2017 at 1:38:42 PM UTC-5, Dave S wrote:
>>>
>>>
>>>
>>> On Monday, January 2, 2017 at 8:22:55 AM UTC-8, Aydin wrote:
>>>>
>>>> I used the exact example shown by massimo (https://vimeo.com/27478796) 
>>>> and it gets stuck at (starting single-scheduler for myapp...) and the 
>>>> website does not come up saying browser cannot establish a connection). 
>>>> Running on ubuntu and debian.
>>>>
>>>
>>> Can you show us your connection string (passwords obfuscated)?
>>>
>>> /dps
>>>  
>>>
>>>>
>>>> On Monday, January 2, 2017 at 11:04:44 AM UTC-5, Niphlod wrote:
>>>>>
>>>>> scheduler definitely works for mysql. 
>>>>> what you experienced is perfectly fine. a scheduler process just sits 
>>>>> there if there is nothing to do (i.e. no tasks to process).
>>>>>
>>>>> On Sunday, January 1, 2017 at 5:14:21 PM UTC+1, Aydin wrote:
>>>>>>
>>>>>> I have used the scheduler before without being worried about the 
>>>>>> database type. 
>>>>>> I tried to use mysql but it gave an error that it is not supported. I 
>>>>>> have mysql and pymysql installed and created a database "testdb" and 
>>>>>> used 
>>>>>> the manual format and got that error that it is not supported.
>>>>>>
>>>>>> On Saturday, December 31, 2016 at 4:09:23 PM UTC-5, 黄祥 wrote:
>>>>>>>
>>>>>>> perhaps, it related with database issue such as : sqlite
>>>>>>> had you already tried another database like mysql or postgresql?
>>>>>>>
>>>>>>> best regards,
>>>>>>> stifan
>>>>>>>
>>>>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Simple scheduler issue

[Niphlod]

scheduler definitely works for mysql. 
what you experienced is perfectly fine. a scheduler process just sits there 
if there is nothing to do (i.e. no tasks to process).

On Sunday, January 1, 2017 at 5:14:21 PM UTC+1, Aydin wrote:
>
> I have used the scheduler before without being worried about the database 
> type. 
> I tried to use mysql but it gave an error that it is not supported. I have 
> mysql and pymysql installed and created a database "testdb" and used the 
> manual format and got that error that it is not supported.
>
> On Saturday, December 31, 2016 at 4:09:23 PM UTC-5, 黄祥 wrote:
>>
>> perhaps, it related with database issue such as : sqlite
>> had you already tried another database like mysql or postgresql?
>>
>> best regards,
>> stifan
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Shared worker

[Niphlod]

no probl at all. the scheduler happily processes tasks from whatever app is 
available. it just needs to be defined in one (and , of course, you need to 
use a unique db for the scheduler throughout your apps)

On Thursday, December 29, 2016 at 8:37:59 PM UTC+1, Gael Princivalle wrote:
>
> Hello.
>
> I've five applications in the same web2py installation.
> For each of these applications I have a set a worker that run task in the 
> scheduler.
> These tasks run one time per day (database csv backup and sitemap).
>
> Is there a way to use one unique worker for these 5 applications?
>
> I would like to have less running processes on the server.
>
> Thanks, regards.
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: column must appear in the GROUP BY clause or be used in an aggregate function

[Niphlod]

once you group, you can only select the grouped fields and any other field 
as an aggregate. 
>From a set-based logic perspective (which is what "serious" database 
enforce) it has absolutely no meaning asking for entity.id in your case.

On Wednesday, December 14, 2016 at 5:44:05 PM UTC+1, Ramos wrote:
>
> can i get some help?
> it was working with sqlite but not with postgresql
>
> for row in db(db.ficheiros.entity==db.entity.id).select(db.entity.entity,
> db.entity.id,count,groupby=db.entity.entity):
>
>
>  column "entity.id" must appear in the 
> GROUP BY clause or be used in an aggregate function LINE 1: SELECT 
> entity.entity, entity.id, COUNT(entity.id) FROM fich... ^
>
> Many thanks
> antónio
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Authomatic Single sign-on with MS Active Directory

[Niphlod]

I took a stab at it but it's incredibly resource-intensive. 
In fact, I just coded a very complicated routine for single sign-on 
(kerberos) in an app that does only user authentication and is used as a 
cas backend by other apps. Restricting the usage to just logging in a few 
times a day is good: trying to put that to authenticate api calls (i.e. 
several times a second) is a no-go.

On Wednesday, December 14, 2016 at 3:06:21 PM UTC+1, Willoughby wrote:
>
> Take a look at the win32security module that's part of pywin32:
> http://timgolden.me.uk/pywin32-docs/win32security.html
>
>
>
> On Wednesday, December 14, 2016 at 8:47:54 AM UTC-5, piero crisci wrote:
>>
>> Hello i am using  correctly the Ldap authentication .
>> But i was wondering if there was a chance to make authomatic sign on 
>> without insert the username and password again.
>> The users infact are on pc on the same domain of the application.
>> Anyone has a hint to help me with this?
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: jquery 1.12.2 vulnerability

[Niphlod]

you are free to use whatever you may like in your own app. 
btw, jquery 1.12.2 afaik is not vulnerable, just "old".

On Monday, December 12, 2016 at 7:53:29 PM UTC+1, Jacinto Parga wrote:
>
>
> Hi, 
>
> I have recently made a security analisys in a web application deployed 
> with web2py in Google App Engine. A jquery 1.12.2 vulnerability was the 
> result:
>
>
> 
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler in systemd environment (running very long tasks)

[Niphlod]

that piece of code is in place to let the worker being terminated by a 
sigterm, i.e a ctrl+c, that is useful for development purposes. it *should* 
have nothing to do with long running tasks, but I'm really honest saying I 
never had a single task alive for more than an hour. Frankly I don't know 
how to test it: being in front of a terminal for 4 days is not that 
feasible.

On Monday, December 12, 2016 at 2:58:47 PM UTC+1, Zbigniew Pomianowski 
wrote:
>
> First of all: I decided to use web2py for my purposes becase it is awesome 
> ;)
> I believe it is not a web2py's bug or anything like related thing. It can 
> be more OS and systemd related issue.
>
> Let me explain what I do and what is the environment. I work in a lab 
> where we try to automate many tests on physical devices (like STBs and 
> phones).
> I have a single source for master (ubuntu server) and slave servers 
> (ubuntu server/desktop). Master is configured with uwsgi+nginx+mysql+web2py 
> services. Then I do have slaves that use the same source, but can spawn 
> tests within scheduler processes.
>
> I need to connect many physical devices to the  slaves (climate chambers, 
> arduino for IR control, v4l2 capture cards, ethernet controled power 
> sources, power supply instruments, measurement instruments... bla bla bla).
> I decided to make a GUI using qooxdoo where user can write a python code 
> that allocates physical devices and run specific test scenarios to examine 
> DUT (Device Under Test) condition.
> These tests sometimes need to be run for tens of hours. So the workflow 
> can be described as:
>
>- user writes a script
>- the test is enqueued as a task in db (JobGraph does a perfect work 
>for me because I need to control the execution sequence mainly because of 
>the existence of physical devices like climate chambers and etc; allocated 
>lab instrument cannot be used by two tests at the same time, jobgraph can 
>yield it) 
>- every slave has it's unique group-name
>   - DUTs and lab instruments are bound to the specific slave - 
>   scheduler group-name
>- slave executes the test scenario programmed by user
>   - test is nothing more than overriden TestUnit
>   - every LAB instrument has child process which logs parameters 
>   (temperature, humidity, voltage bla bla bla)
>   - for DUT is also created instance of a class that spawns child 
>   processes (video freeze detection based on gstreamer, udp/tcp/telnet 
>   interface to interract with STB)
>   - in test scenario I have plenty of sleeps - test scenario demands 
>   for example that STB stays in a cimate chamber for 20h in specific temp 
> and 
>   humidity
>
> My systemd service file looks like this:
> [Unit]
> Description=ATMS workers
> After=network-online.target
> Wants=network-online.target
>
> [Service]
> User=
> Restart=on-failure
> RestartSec=120
> Environment=DISPLAY=: # usually 0
> Environment=XAUTHORITY=/home//.Xauthority
> EnvironmentFile={{INSTALL}}/web2py_venv/web2py/applications/atms/private/
> atms.env
> ExecStartPre=/bin/sh -c "${WEB2PYPY} ${WEB2PY} -S atms -M -R 
> ${WEB2PYDIR}/applications/atms/systemd/on_start.py -P"
> ExecStart=/bin/sh -c "${WEB2PYPY} ${WEB2PY} -K atms:%H,atms:%H"
> ExecStop=/bin/sh -c "${WEB2PYPY} ${WEB2PY} -S atms -M -R 
> ${WEB2PYDIR}/applications/atms/systemd/on_stop.py -P"
>
> [Install]
> # graphical because i had to make some kind of preview with ximagesink for 
> fast lookup if video is ok on STB
> WantedBy=graphical.target
> Alias=atms.service
>
>
> I realised that for very long test (last one was planned to be longer than 
> 100h) i got  sth like this in logs:
> gru 11 12:01:52 slaveX sh[2184]:   File 
> "/atms/web2py_venv/web2py/gluon/packages/dal/pydal/adapters/base.py", 
> line 1435, in
> gru 11 12:01:52 slaveX sh[2184]: return str(long(obj))
> gru 11 12:01:52 slaveX sh[2184]:   File 
> "/atms/web2py_venv/web2py/gluon/packages/dal/pydal/objects.py", line 82, 
> in  gru 11 12:01:52 slaveX sh[2184]: __long__ = lambda self: long(self.get
> ('id'))
> gru 11 12:01:52 slaveX sh[2184]: TypeError: long() argument must be a 
> string or a number, not 'NoneType'
>
> The test was stopped 20h before it was supposed to be finished :/
> After some digging I found that before these errors i got this one:
> gru 11 12:01:34 slaveX sh[2184]: ERROR:web2py.app.atms:[( taskId10672_caseId852_duts32/test_script.py.TestCase testMethod=
> test_example>, 'Traceback (most recent call last):\n  File 
> "/tmp/taskId10672_caseId852_duts32/test_script.py", line 90, in 
> test_example\nsleep(M10)\n  File 
> "/atms/web2py_venv/web2py/gluon/scheduler.py", line 702, in \n   
>  signal.signal(signal.SIGTERM, lambda signum, stack_frame: 
> sys.exit(1))\nSystemExit: 1\n')]
> gru 11 12:01:34 slaveX sh[2184]: DEBUG:web2py.app.atms:new task report
> : FAILED
> gru 11 12:01:34 slaveX sh[2184]: DEBUG:web2py.app.atms:   traceback: 
> Traceback (most recent call 

[web2py] Re: Contributing to web2py

[Niphlod]

- if you cloned from github without reading the main page  I wonder 
what "documentation" needs to be more explicit.
- you can single tests with run python -m unittest gluon.tests.whatever


On Monday, December 12, 2016 at 12:20:26 AM UTC+1, Marlysson Silva wrote:
>
> Guys , good night .
>
> I made a clone of repository of web2py to add some codes and i had some 
> issues..
>
> 1. Need clone a submodule join with web2py's main repository ( ok isn't a 
> issue , just isn't explicit a lot in documentation );
>
> 2. I added my code improvement code and ran the tests with command :
> python web2py.py --run_system_tests
>
> This case ran all tests from system.. There some way to execute individual 
> test by file? E.g:
> python -m unittest test_utils.py
> python test_utils.py
>
> 3. And at end of all tests yet exists errors:
>
> Ran 311 tests in 189.749s
>
> FAILED (failures=1, errors=4, skipped=1)
>
>
> Some errors of encoding and index out of range..
>
> It my machine or the tests are wrong same..
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: Scheduler task every first day of every month ?

[Niphlod]

this is an already-solved problem. fetch scheduler.py from the master repo: 
you can schedule recurrences using the cronline parameter.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: build bridging scanner and Active Directory with scanner API (xml)

[Niphlod]

it's still absolutely NOT clear what you need to do. "bridge a scanner with 
LDAP" is quite vague

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Syntax question: db(db.mytable.id == request.post_vars["id"]).update(**{key:value})

[Niphlod]

I'd loop collecting the needed updates in a dict and then issue a single 
update at the end with the **dict

On Tuesday, December 6, 2016 at 11:44:56 PM UTC+1, Henk huisman wrote:
>
> Is this correct syntax?
>
> if request.post_vars["id"]:
>> for key, value in request.post_vars.items(): 
>>   db(db.mytable.id == request.post_vars["id"]).update(**{key:value})
>>
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: db Field unique=True

[Niphlod]

web2py forms use postbacks (the submit is directed to the same address that 
generated it) so the general flow is the following:

- user hits the page
- the form was not submitted nor accepted nor errored (goes to the last 
else)
- user fills the forms and click submit
- form was submitted: it can be accepted or errored (goes to if 
form.accepted or if form.errors)

>From the moment user submitted the form that errored to the form rightfully 
compiled there is no shared state.
If you really need what you're asking (I'm quite sure that your logic may 
be pleased with e.g. generate a random uid just in the "accepted" branch) 
you need to alter the logic: persist the need of generating a new uid 
somewhere, i.e. the session, and then alter the default value for uid 
accordingly.
Or just make your own db I/O with e.g. SQLFORM.factory .

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: build bridging scanner and Active Directory with scanner API (xml)

[Niphlod]

why not ? what's your problem ?

On Tuesday, December 6, 2016 at 9:32:31 PM UTC+1, 黄祥 wrote:
>
> is it possible to build bridging scanner and Active Directory with scanner 
> API (xml) using web2py?
> *e.g. xml for Menu to choosing system*
> 
> 
> 
> 
>  
> 
> 
> 
> 
> Scan
> 
> 
> Copy
> 
> 
> 
> 
> ]]>
> 
> 
> 
>
> *e.g. xml for copy*
> 
> 
> 
> Auto
> Normal
> 100
> Normal
> Landscape
> true
> 1
> ./selesai.xml
> 
> 
> 
>  
> 
> 
> 
> executing copy
> 
> 
> 
> ]]>
> 
> 
> 
>
> *e.g. xml for FTP command*
> 
> 
> 
> 
> 
> 
> test
> example.com
> usertest
> 123456789
> /
> true
> 21
> 
> 
> 
> 
> Gray
> Low
> 
> PDF
> ./done.xml
> 
> 
> 
>  
> 
> 
> 
> executing scan2ftp
> 
> 
> 
> ]]>
> 
> 
> 
>
> thanks and best regards,
> stifan
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Syntax for placeholders in executesql

[Niphlod]

apparently it's :1, :2, etc etc etc

see the end of 
http://www.oracle.com/technetwork/articles/dsl/prez-python-queries-101587.html



On Tuesday, December 6, 2016 at 6:58:01 PM UTC+1, Ty oc wrote:
>
> By the way, what case would be the xintaxis for oracle as database?
>
> El martes, 6 de diciembre de 2016, 11:46:34 (UTC-6), Ty oc escribió:
>>
>> Well, if this is so specific, why it isn't documented with the current 
>> simple equivalent examples for each supported driver??
>>
>>
>> I mean
>>
>> placeholders is an optional sequence of values to be substituted in or, 
>>> if supported by the DB driver, a dictionary with keys matching named 
>>> placeholders in your SQL.
>>>
>>
>> Says almost nothing at all.
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: switch to python 3

[Niphlod]

as I said, definitely not stable nor released.

On Tuesday, December 6, 2016 at 7:34:09 PM UTC+1, Paolo Valleri wrote:
>
> What about https://gitlab.com/m2crypto/m2crypto/commits/python3 ?
>
>
> On Monday, December 5, 2016 at 9:57:19 PM UTC+1, Niphlod wrote:
>>
>> m2crypto port for py3 are available (although not strictly official). 
>> The contrib module for x509 auth stands on it so, on py3, it MAY not 
>> work, but it's not web2py's fault (BTW, if you have an alternative, that 
>> works for py2 and py3 we'll be glad to adopt it)
>>
>>
>> On Monday, December 5, 2016 at 1:31:19 AM UTC+1, Jason Solack wrote:
>>>
>>> Does not having m2crypto mean auth is less secure or does it just effect 
>>> x509 authentication?
>>>
>>>
>>> On Sunday, November 6, 2016 at 2:15:08 PM UTC-5, Niphlod wrote:
>>>>
>>>> the same gain moving any piece of code from python 2 to python 3 (read 
>>>> on the interwebs, there are a few).
>>>> at least now you can choose and have web2py not standing in the way.
>>>>
>>>> On Sunday, November 6, 2016 at 12:34:39 AM UTC+1, Ramos wrote:
>>>>>
>>>>> what is the gain moving to python 3?
>>>>>
>>>>> 2016-11-05 17:59 GMT+00:00 Leonel Câmara <leonel...@gmail.com>:
>>>>>
>>>>>> web2py is almost completely python 3 compatible. You can follow the 
>>>>>> issue here: https://github.com/web2py/web2py/issues/1353
>>>>>>
>>>>>> -- 
>>>>>> Resources:
>>>>>> - http://web2py.com
>>>>>> - http://web2py.com/book (Documentation)
>>>>>> - http://github.com/web2py/web2py (Source code)
>>>>>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>>>>> --- 
>>>>>> You received this message because you are subscribed to the Google 
>>>>>> Groups "web2py-users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>> send an email to web2py+un...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: Moving from sqlite to postgresql - operator does not exist: integer = text

[Niphlod]

fixing your import process and/or correcting the wrong records.

On Tuesday, December 6, 2016 at 10:43:14 AM UTC+1, Ramos wrote:
>
> I exported to csv all major tables than imported them to my new app via 
> admin .
> I think i found the bug
> its here 
> &(db.status.id==db.events.event)
> status.id is integer but db.events.event is a text field that contains 
> "1"
> "2"
> "3"
> "4"
> "sometext"
> "anothertext"
> "6"
> "7"
> "8"
>
> How do i solve this?
>
> Regards
> António
>
>
> 2016-12-05 20:49 GMT+00:00 Niphlod <nip...@gmail.com >:
>
>> just check that all your columns where you do == have the same type. 
>> sqlite datatype handling is more resilient than standard, meaning that 
>> there is no strict type check.
>> dunno how you migrated data from sqlite to postgresql but probably 
>> something doesn't match .
>>
>> -- 
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to web2py+un...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: admin interface

[Niphlod]

this is far clearer. 

If you want a single web2py instance to behave like that, you NEED to use 
routes.py, enabling/disabling routes to the admin app inspecting the 
hostname it's called in.

OR, you can just short-circuit admin adding a model file (name it 
alphabetically so it gets executed before all others) preventing access 
from any hostname doesn't match the one you want.



On Monday, December 5, 2016 at 7:57:22 AM UTC+1, Annet wrote:
>
> Hi Anthony,
>
> Thanks for your reply.
>
> I apologize for not being clear enough,
>
> I have one installation of web2py and 4 applications each being referenced 
> by their own domain.
>
> dbadmin by admin.mydomain.com
> init by www.mydomain,com
> etc.
>
> I only want acces the web2py interface via admin.mydomain.com not via the 
> other domains.
> However, at the moment I can acces the web2py webinterface via:
>
> https://admin.mydomain.com.nl/admin?send=%2Fadmin%2Fdefault%2Fsite
>
> https://www..mydomain.com.nl/admin?send=%2Fadmin%2Fdefault%2Fsite
>
> https://cms.mydomain.com.nl/admin?send=%2Fadmin%2Fdefault%2Fsite
>
> I want the admin interface disabled for all domain accept 
> admin.mydomain.com
>
>
> Kind regards,
>
> Annet
>
>
> On Friday, 2 December 2016 17:25:09 UTC+1, Anthony wrote:
>>
>> On Friday, December 2, 2016 at 9:29:00 AM UTC-5, Annet wrote:
>>>
>>> Hi,
>>>
>>> Thanks for your reply. You mean an empty .py file named DISABLED
>>>
>>
>> That will completely disable the app, not prevent it from showing up in 
>> admin.
>>  
>>
>>>
>>> What does ATM stand for?
>>>
>>
>> "at the moment," probably.
>>
>> Anthony
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: switch to python 3

[Niphlod]

m2crypto port for py3 are available (although not strictly official). 
The contrib module for x509 auth stands on it so, on py3, it MAY not work, 
but it's not web2py's fault (BTW, if you have an alternative, that works 
for py2 and py3 we'll be glad to adopt it)


On Monday, December 5, 2016 at 1:31:19 AM UTC+1, Jason Solack wrote:
>
> Does not having m2crypto mean auth is less secure or does it just effect 
> x509 authentication?
>
>
> On Sunday, November 6, 2016 at 2:15:08 PM UTC-5, Niphlod wrote:
>>
>> the same gain moving any piece of code from python 2 to python 3 (read on 
>> the interwebs, there are a few).
>> at least now you can choose and have web2py not standing in the way.
>>
>> On Sunday, November 6, 2016 at 12:34:39 AM UTC+1, Ramos wrote:
>>>
>>> what is the gain moving to python 3?
>>>
>>> 2016-11-05 17:59 GMT+00:00 Leonel Câmara <leonel...@gmail.com>:
>>>
>>>> web2py is almost completely python 3 compatible. You can follow the 
>>>> issue here: https://github.com/web2py/web2py/issues/1353
>>>>
>>>> -- 
>>>> Resources:
>>>> - http://web2py.com
>>>> - http://web2py.com/book (Documentation)
>>>> - http://github.com/web2py/web2py (Source code)
>>>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>>> --- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "web2py-users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to web2py+un...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Javascript functions for Canvas not working

[Niphlod]

it'd be hardly an issue with bootstrap itself given bootstrap "share" among 
websites

On Friday, December 2, 2016 at 9:38:09 PM UTC+1, Dave S wrote:
>
>
>
> On Friday, December 2, 2016 at 11:55:33 AM UTC-8, Brian M wrote:
>>
>> In case others have this sort of a problem where javascript events are 
>> affected by the css of bootstrap...
>>
>> The problem was in the css file:  bootstrap.min.css
>>
>> Somehow the relative position causes the event not to bubble up. How 
>> jacked up is that?
>>
>> .col-xs-1,.col-sm-1,.col-md-1,.col-lg-1,.col-xs-2,.col-sm-2,.col-md-2,.col-lg-2,.col-xs-3,.col-sm-3,.col-md-3,.col-lg-3,.col-xs-4,.col-sm-4,.col-md-4,.col-lg-4,.col-xs-5,.col-sm-5,.col-md-5,.col-lg-5,.col-xs-6,.col-sm-6,.col-md-6,.col-lg-6,.col-xs-7,.col-sm-7,.col-md-7,.col-lg-7,.col-xs-8,.col-sm-8,.col-md-8,.col-lg-8,.col-xs-9,.col-sm-9,.col-md-9,.col-lg-9,.col-xs-10,.col-sm-10,.col-md-10,.col-lg-10,.col-xs-11,.col-sm-11,.col-md-11,.col-lg-11,.col-xs-12,.col-sm-12,.col-md-12,.col-lg-12{position:relative;min-height:1px;padding-left:15px;padding-right:15px}
>>
>>
>>
>>
> Interesting.   Massimo has examples of web2py usage that don't use 
> bootstrap ... he's worked up stupd.css for these;  the bundled app at 
> web2py/application/examples is the online book using stupid.css.  There are 
> also some sample pages as part of the github for stupid.css 
> 
>
> I don't know how much you're committed to the current views, or if you 
> still have the option of trying the lighter css.  But you can check it out 
> if you want.
>
> /dps
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Moving from sqlite to postgresql - operator does not exist: integer = text

[Niphlod]

just check that all your columns where you do == have the same type. 
sqlite datatype handling is more resilient than standard, meaning that 
there is no strict type check.
dunno how you migrated data from sqlite to postgresql but probably 
something doesn't match .

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: admin interface

[Niphlod]

you open /admin ?!

why is that an issue ? 

BTW: for disabling ATM you can just put a "DISABLED" file in the root 
folder of the app.

On Friday, December 2, 2016 at 11:38:56 AM UTC+1, Annet wrote:
>
> I have an application with limited functionality in which I want to access
> web2py's admin interface.
>
> I have 3 application in the same web2py installation in which I want to
> disable the admin interface.
>
> Is this possible and how would I do this.
>
>
> Kind regards,
>
> Annet
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: import issues

[Niphlod]

simplejson was dropped, as we target 2.7 only and there's the json module 
globally available. you should use that.

I **think** this still works, but it's the only thing permitted.


from gluon.contrib.simplejson import json



On Friday, December 2, 2016 at 2:37:36 AM UTC+1, Dave S wrote:
>
> As a move to use tcpdump to look at the actual raw response (my issue 
> where it appears I need to force an ascii response),
> I copied my controller code from my dev machine to a test machine.  And 
> ran into import errors trying to import simplejson. (The ticket says it was 
> trying to load applications.myapp.modules.simplejson, so I tried changing 
> it with a "from contrib.simplejson.simplejson" -- oops, there's no 
> simplejson.py in gluon/contrib/simplejson, so that still errors (different 
> ticket).
>
> What am I missing?  I'm seeing this on both 2.14.4 and 2.14.6 on the test 
> machine (which has been running the other functions of the controller for 
> ages and ages).
>
> /dps
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Action commit and scheduled task commit racing?

[Niphlod]

It absolutely can't happen.
But the logic in the code is quite prone to inconveniences: every time a task 
runs, it only processes the last record of the table...what happens if you have 
called the webhook function several times before the task has the chance to be 
executed? You end up having holes of processed webhooks

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Action commit and scheduled task commit racing?

[Niphlod]

It absolutely can't happen.
But the logic in the code is quite prone to inconveniences: every time a task 
runs, it only processes the last record of the table...what happens if you have 
called the webhook function several times before the task has the chance to be 
executed? You end up having holes of processed webhooks

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Action commit and scheduled task commit racing?

[Niphlod]

It absolutely can't happen.
But the logic in the code is quite prone to inconveniences: every time a task 
runs, it only processes the last record of the table...what happens if you have 
called the webhook function several times before the task has the chance to be 
executed? You end up having holes of processed webhooks

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: Scheduler error

[Niphlod]

BTW: you could be facign this issue that was resolved on master

https://github.com/web2py/web2py/issues/1485



On Wednesday, November 23, 2016 at 6:58:03 PM UTC+1, Niphlod wrote:
>
> it's very simple then: if it's failed , the traceback is stored in the 
> corresponding scheduler_run table !
>
> On Wednesday, November 23, 2016 at 3:44:03 PM UTC+1, Ramos wrote:
>>
>> The task itself is set as failed!
>>
>>
>>
>> Enviado a partir do meu smartphone Samsung Galaxy.
>>
>>  Mensagem original 
>> De: Niphlod <nip...@gmail.com> 
>> Data: 23/11/16 13:08 (GMT+00:00) 
>> Para: web2py-users <web...@googlegroups.com> 
>> Assunto: [web2py] Re: Scheduler error 
>>
>> when you say "fail" you mean the status of the task itelf is set as 
>> FAILED or that your db.emails.status is 'failed' ?
>>
>> On Wednesday, November 23, 2016 at 1:13:10 PM UTC+1, Ramos wrote:
>>>
>>> web2py 2.14.3
>>>
>>> 2016-11-23 12:07 GMT+00:00 António Ramos <ramst...@gmail.com>:
>>>
>>>> i think the problem is in response.render... because i change it to a 
>>>> simple email with no template rendering and it works..
>>>>
>>>> 2016-11-23 11:51 GMT+00:00 António Ramos <ramst...@gmail.com>:
>>>>
>>>>> what can be wrong with the code in my scheduler.py ?
>>>>> *montlyreport* is called by a task with the same name
>>>>> I have 2 users in auth_user table
>>>>> I see 2 emails generated in table "emails"
>>>>> then 2 tasks generated in scheduler_task
>>>>> But these 2 tasks fail and no info about what failed...
>>>>> Regards
>>>>> [image: Imagem inline 2]
>>>>>
>>>>
>>>>
>>> -- 
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to web2py+un...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: 502 Bad Gateway - no tmp/web2py.socket file

[Niphlod]

okay, now we have a deal ^_^ ...

what about 

journalctl -u emperor.uwsgi.service 

?

On Wednesday, November 23, 2016 at 2:27:20 PM UTC+1, Richard Brown wrote:
>
> Hi, there are no files in '/var/log/uwsgi' and  "journalctl 
> /usr/local/bin/uwsgi" reports "No journal files were found."
>
>
> On Wednesday, November 23, 2016 at 1:14:00 PM UTC, Niphlod wrote:
>
>> okay, you can see in the first line that /tmp/web2py.socket is mentioned.
>>
>> if uwsgi is running, need to check the log to see if any errors comes up
>>
>> check in /var/log/uwsgi and/or see if systemd logging reports something 
>> with "journalctl /usr/local/bin/uwsgi"
>>
>> On Wednesday, November 23, 2016 at 1:36:35 PM UTC+1, Richard Brown wrote:
>>>
>>> Thanks, the answer is 'yes' to all of these checks - web2py.ini file 
>>> attached. I did find out how to see if the uwsgi process is running and it 
>>> does appear to be:
>>>
>>>  $ ps aux | grep "uwsgi"
>>> richard   1044  0.0  0.1   4276  1824 pts/0S+   10:54   0:00 grep 
>>> --color=auto uwsgi
>>>
>>>
>>>
>>> On Wednesday, November 23, 2016 at 11:54:21 AM UTC, Niphlod wrote:
>>>
>>>> ok, let's break it down.
>>>> The script basically does:
>>>> - install and configure nginx
>>>> - install and configure uwsgi
>>>>
>>>> uwsgi is the process that takes care of executing any python code 
>>>> (meaning, your webapp) .
>>>> nginx is the process that takes care of receiving an http request, 
>>>> passing it down to uwsgi, get the results and send the http response back 
>>>> to your browser.
>>>> Both need to be running in order for your app to work.
>>>>
>>>> In order to let the two communicate, nginx expects a "file" 
>>>> (/tmp/web2py.socket) . That file is created by uwsgi.
>>>> It's possible that one of the steps of the script didn't work out, 
>>>> hence your error.
>>>> The error you posted seems to point out that nginx is configured 
>>>> correctly, but for some reason uwsgi is not.
>>>>
>>>> The first thing to check is that you have the uwsgi daemon.
>>>> There's a command, "which", that will tell you the location of any 
>>>> executable is ready on your system.
>>>> So, type  "which uwsgi" in a terminal to know if at least it got 
>>>> installed. it should reply "/usr/local/bin/uwsgi"
>>>>
>>>> If it's installed, the next check is see if it's configured correctly.
>>>> Systemd is used to let uwsgi start:
>>>> - Check if you have a file at /etc/systemd/system/emperor.uwsgi.service
>>>> An ini file is responsible to let uwsgi know WHAT to run:
>>>> - Check if you have a file at /etc/uwsgi/web2py.ini
>>>>
>>>> If all of the above checks out, we'll need further steps, but for the 
>>>> time being check them 
>>>>
>>>>
>>>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] how to quit ajax LOAD loop

[Niphlod]

if you add "w2p_component_stop" as a class to the element, the cycle will 
be interrupted.

On Wednesday, November 23, 2016 at 4:50:47 PM UTC+1, Richard wrote:
>
> Did you try ajax_trap=False
>
> Or you can redirect by specifying the extension='html' if it does work, so 
> you can redirect once form is accepted.
>
> Richard
>
> On Wed, Nov 23, 2016 at 9:50 AM, Pierre  > wrote:
>
>> Hi,
>>
>> i wrote this in a view :
>>
>>  
>> {{=LOAD('default','ustat.load', args=[k], target='pcustat',
>> ajax=True,times=10, timeout=5000, 
>> content=T('loading...'))}}
>> 
>>
>> I want to quit the LOAD loop when *ustat *returns 'OK'
>>
>> response.js = "jQuery('#pcustat').remove();" 
>>
>> generates a web2py.js jquery get error and i also would like to keep 
>> *ustat* returned value within the?
>>
>> -- 
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to web2py+un...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: Scheduler error

[Niphlod]

it's very simple then: if it's failed , the traceback is stored in the 
corresponding scheduler_run table !

On Wednesday, November 23, 2016 at 3:44:03 PM UTC+1, Ramos wrote:
>
> The task itself is set as failed!
>
>
>
> Enviado a partir do meu smartphone Samsung Galaxy.
>
>  Mensagem original 
> De: Niphlod <nip...@gmail.com > 
> Data: 23/11/16 13:08 (GMT+00:00) 
> Para: web2py-users <web...@googlegroups.com > 
> Assunto: [web2py] Re: Scheduler error 
>
> when you say "fail" you mean the status of the task itelf is set as FAILED 
> or that your db.emails.status is 'failed' ?
>
> On Wednesday, November 23, 2016 at 1:13:10 PM UTC+1, Ramos wrote:
>>
>> web2py 2.14.3
>>
>> 2016-11-23 12:07 GMT+00:00 António Ramos <ramst...@gmail.com>:
>>
>>> i think the problem is in response.render... because i change it to a 
>>> simple email with no template rendering and it works..
>>>
>>> 2016-11-23 11:51 GMT+00:00 António Ramos <ramst...@gmail.com>:
>>>
>>>> what can be wrong with the code in my scheduler.py ?
>>>> *montlyreport* is called by a task with the same name
>>>> I have 2 users in auth_user table
>>>> I see 2 emails generated in table "emails"
>>>> then 2 tasks generated in scheduler_task
>>>> But these 2 tasks fail and no info about what failed...
>>>> Regards
>>>> [image: Imagem inline 2]
>>>>
>>>
>>>
>> -- 
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> --- 
> You received this message because you are subscribed to the Google Groups 
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to web2py+un...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: 502 Bad Gateway - no tmp/web2py.socket file

[Niphlod]

okay, you can see in the first line that /tmp/web2py.socket is mentioned.

if uwsgi is running, need to check the log to see if any errors comes up

check in /var/log/uwsgi and/or see if systemd logging reports something 
with "journalctl /usr/local/bin/uwsgi"

On Wednesday, November 23, 2016 at 1:36:35 PM UTC+1, Richard Brown wrote:
>
> Thanks, the answer is 'yes' to all of these checks - web2py.ini file 
> attached. I did find out how to see if the uwsgi process is running and it 
> does appear to be:
>
>  $ ps aux | grep "uwsgi"
> richard   1044  0.0  0.1   4276  1824 pts/0S+   10:54   0:00 grep 
> --color=auto uwsgi
>
>
>
> On Wednesday, November 23, 2016 at 11:54:21 AM UTC, Niphlod wrote:
>
>> ok, let's break it down.
>> The script basically does:
>> - install and configure nginx
>> - install and configure uwsgi
>>
>> uwsgi is the process that takes care of executing any python code 
>> (meaning, your webapp) .
>> nginx is the process that takes care of receiving an http request, 
>> passing it down to uwsgi, get the results and send the http response back 
>> to your browser.
>> Both need to be running in order for your app to work.
>>
>> In order to let the two communicate, nginx expects a "file" 
>> (/tmp/web2py.socket) . That file is created by uwsgi.
>> It's possible that one of the steps of the script didn't work out, hence 
>> your error.
>> The error you posted seems to point out that nginx is configured 
>> correctly, but for some reason uwsgi is not.
>>
>> The first thing to check is that you have the uwsgi daemon.
>> There's a command, "which", that will tell you the location of any 
>> executable is ready on your system.
>> So, type  "which uwsgi" in a terminal to know if at least it got 
>> installed. it should reply "/usr/local/bin/uwsgi"
>>
>> If it's installed, the next check is see if it's configured correctly.
>> Systemd is used to let uwsgi start:
>> - Check if you have a file at /etc/systemd/system/emperor.uwsgi.service
>> An ini file is responsible to let uwsgi know WHAT to run:
>> - Check if you have a file at /etc/uwsgi/web2py.ini
>>
>> If all of the above checks out, we'll need further steps, but for the 
>> time being check them 
>>
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Scheduler error

[Niphlod]

when you say "fail" you mean the status of the task itelf is set as FAILED 
or that your db.emails.status is 'failed' ?

On Wednesday, November 23, 2016 at 1:13:10 PM UTC+1, Ramos wrote:
>
> web2py 2.14.3
>
> 2016-11-23 12:07 GMT+00:00 António Ramos  >:
>
>> i think the problem is in response.render... because i change it to a 
>> simple email with no template rendering and it works..
>>
>> 2016-11-23 11:51 GMT+00:00 António Ramos > >:
>>
>>> what can be wrong with the code in my scheduler.py ?
>>> *montlyreport* is called by a task with the same name
>>> I have 2 users in auth_user table
>>> I see 2 emails generated in table "emails"
>>> then 2 tasks generated in scheduler_task
>>> But these 2 tasks fail and no info about what failed...
>>> Regards
>>> [image: Imagem inline 2]
>>>
>>
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Processing taking a long time in web2py?

[Niphlod]

try to run ti inside a scheduler task

On Wednesday, November 23, 2016 at 1:55:01 PM UTC+1, Si wrote:
>
> Thanks for the reply.
> I understand that they are different things. I was wondering if you could 
> point me in the right direction to somehow be able to speed this up? Or at 
> least find out where the time is being spent. Web2py adding ~20 seconds is 
> quite a lot of time.
> In the future I will be using a library, but for now I have to use this 
> external process.
>
>
> On Sunday, 20 November 2016 20:47:33 UTC, Niphlod wrote:
>>
>> hiding behind auth doesn't get you protected by DDoS. 
>> opening a process inside a web2py app vs pure python depends on you 
>> comparing oranges to apples. web2py env with the default webserver is a 
>> threaded env, your pure python process probably isn't.
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: 502 Bad Gateway - no tmp/web2py.socket file

[Niphlod]

ok, let's break it down.
The script basically does:
- install and configure nginx
- install and configure uwsgi

uwsgi is the process that takes care of executing any python code (meaning, 
your webapp) .
nginx is the process that takes care of receiving an http request, passing 
it down to uwsgi, get the results and send the http response back to your 
browser.
Both need to be running in order for your app to work.

In order to let the two communicate, nginx expects a "file" 
(/tmp/web2py.socket) . That file is created by uwsgi.
It's possible that one of the steps of the script didn't work out, hence 
your error.
The error you posted seems to point out that nginx is configured correctly, 
but for some reason uwsgi is not.

The first thing to check is that you have the uwsgi daemon.
There's a command, "which", that will tell you the location of any 
executable is ready on your system.
So, type  "which uwsgi" in a terminal to know if at least it got installed. 
it should reply "/usr/local/bin/uwsgi"

If it's installed, the next check is see if it's configured correctly.
Systemd is used to let uwsgi start:
- Check if you have a file at /etc/systemd/system/emperor.uwsgi.service
An ini file is responsible to let uwsgi know WHAT to run:
- Check if you have a file at /etc/uwsgi/web2py.ini

If all of the above checks out, we'll need further steps, but for the time 
being check them 


-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: How to create RSS feed from internal table?

[Niphlod]

see 
https://github.com/web2py/web2py/blob/master/applications/welcome/views/generic.rss

On Wednesday, November 23, 2016 at 8:20:20 AM UTC+1, Alex Glaros wrote:
>
> I have a web2py table, db.meeting, whose records are to be consumed by 
> outside parties as an RSS feed.
>
> In other words, people will subscribe to the RSS feed to obtain a list of 
> meeting dates, times, locations and subjects.
>
> Does anyone have an example of the syntax?  The examples in the book seem 
> to use external, already-existing RSS data that is to be aggregated.
>
> I want to to create the data from internal web2py tables.
>
> thanks
>
> Alex Glaros
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: 502 Bad Gateway - no tmp/web2py.socket file

[Niphlod]

one step deployment imho is not "assured to be working" on raspberry. 
easy check: do you have an uwsgi process running ?

On Tuesday, November 22, 2016 at 11:19:32 PM UTC+1, Richard Brown wrote:
>
> Thanks for the reply. The key word here is obviously 'should' - but is it 
> something that should have happened or something I should have done, and 
> how do I resolve it? I used the 'one step deployment' script as thought it 
> would do everything I need to get Web2py working.
>
> Richard
>
> On Tuesday, November 22, 2016 at 7:19:25 PM UTC, Niphlod wrote:
>
>> that should be generated by the uwsgi daemon
>>
>> On Tuesday, November 22, 2016 at 6:34:40 PM UTC+1, Richard Brown wrote:
>>>
>>> I've just tried to do a new install of Web2py on a Raspberry Pi 3 having 
>>> been using it under Apache on a Model B for a few years now. I used the 
>>> script "setup-web2py-nginx-uwsgi-ubuntu.sh" which seems to have done most 
>>> things correctly, but when I try to access the site I get a '502 Bad 
>>> Gateway' error in my browser and the following error in 
>>> /var/log/nginx/error.log:
>>>
>>> 2016/11/22 16:41:30 [crit] 648#0: *1 connect() to 
>>> unix:///tmp/web2py.socket failed (2: No such file or directory) while 
>>> connecting to upstream, client: 192.168.1.65, server: raspberrypi, request: 
>>> "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: 
>>> "192.168.1.60"
>>>
>>> There is no web2py.socket file in the /tmp directory or anywhere else 
>>> that I can find. I can see where the script sets up the path to this file 
>>> but nothing about how the file itself should get there. Where do I find 
>>> this file, or should the configuration point to a different file?
>>>
>>> Thanks,
>>>
>>> Richard
>>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: 502 Bad Gateway - no tmp/web2py.socket file

[Niphlod]

that should be generated by the uwsgi daemon

On Tuesday, November 22, 2016 at 6:34:40 PM UTC+1, Richard Brown wrote:
>
> I've just tried to do a new install of Web2py on a Raspberry Pi 3 having 
> been using it under Apache on a Model B for a few years now. I used the 
> script "setup-web2py-nginx-uwsgi-ubuntu.sh" which seems to have done most 
> things correctly, but when I try to access the site I get a '502 Bad 
> Gateway' error in my browser and the following error in 
> /var/log/nginx/error.log:
>
> 2016/11/22 16:41:30 [crit] 648#0: *1 connect() to 
> unix:///tmp/web2py.socket failed (2: No such file or directory) while 
> connecting to upstream, client: 192.168.1.65, server: raspberrypi, request: 
> "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: 
> "192.168.1.60"
>
> There is no web2py.socket file in the /tmp directory or anywhere else that 
> I can find. I can see where the script sets up the path to this file but 
> nothing about how the file itself should get there. Where do I find this 
> file, or should the configuration point to a different file?
>
> Thanks,
>
> Richard
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: why DAL decode_credentials=True can't work for task scheduler worker?

[Niphlod]

scheduler just accepts a DAL connection as a parameter. 
There's no way that db connection can work for the web process and not the 
scheduler one.

On Monday, November 21, 2016 at 4:42:55 PM UTC+1, DengFeng Mao wrote:
>
> I am using web2py Version 2.14.3-stable+timestamp.2016.03.26.23.02.02 with 
> Mysql
> There is '@' in mysql password, Then I set  decode_credentials=True in DAL.
> The web access and mysql access work well.
> Then I tried to add a task scheduler. The db table 'db.scheduler_task' , 
> 'db.scheduler_run', 'db.scheduler_worker' tables were created automatically 
> by the database migration.
> Question was coming when I tried to start a worker process with the 
> following command:
>  ./web2py  -K  myapp
>
> The error like this:
> -
> RuntimeError: Failure to connect, tried 5 times:
> OperationalError: (1045, u"Access denied for user   ...
> ---
>
> It confused me. There is no any issue for the web access database.  But 
> scheduler worker process can't connect database.
> Why?  I had thought  both web and scheduler worker use the same DAL 
> connection
>
> I tired mysql password without '@' and  DAL decode_credentials=False.
> scheduler worker process can connect database as expectation. But it 
> complaint
> ---
> InternalError: (1050, u"Table 'auth_user' already exists")
> 
>
> I know , I should set 'migrate=False' for the table 'auth_user'. But I 
> wonder why the web access process didn't complain that error?
> Only scheduler worker process complaint  it
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: what is in the cache ?

[Niphlod]

in redis terms (which will set an expire timing accordingly) doesn't make 
much sense. 

In the default implementation, which never expires anything, time_expire=0 
basically means "update that key".

On Monday, November 21, 2016 at 4:30:07 PM UTC+1, Pierre wrote:
>
> I understand  use-cases for 
> time_expire=None
>
>
> but I don't see a use-case for 
> time_expire=0
>
>
>
> when you do  
> cache.redis('message', lambda: 'Hello', time_expire=0)
>
> all the subsequent *cache.redis('message',)* will overwrite Hello
>
> to get a persistent Hello you have to write something like:
>
> cache.redis('message', None)
> cache.redis('message',lambda: 'Hello', time_expire=None)
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: what is in the cache ?

[Niphlod]

goes to redis, obviously! and redis usually stores it into ram (although it 
usually saves a copy to the disk, which is configurable, but just to not 
make it totally volatile between restarts of redis)

On Friday, November 18, 2016 at 12:21:05 PM UTC+1, Pierre wrote:
>
> Yes I improved my understanding of the cache system since the last 
> message. now writing a component to make sure i can integrate this 
> thing..
>
> where does *cache.redis* goes : ram or disk ?
>
> thanks
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Processing taking a long time in web2py?

[Niphlod]

hiding behind auth doesn't get you protected by DDoS. 
opening a process inside a web2py app vs pure python depends on you 
comparing oranges to apples. web2py env with the default webserver is a 
threaded env, your pure python process probably isn't.


On Friday, November 18, 2016 at 8:34:46 AM UTC+1, Si An wrote:
>
> I'm asking why opening a process in a python script takes much less time 
> than opening the same python code/process in a web2py application.
>
> Also yes I will consider your second point but I think I will hide it 
> behind authentication
>
> On Thursday, 17 November 2016 22:53:50 UTC, Niphlod wrote:
>>
>> well... you're asking why opening a process in python rather than doing 
>> it directly takes more time. no wonders there: of course it takes more 
>> time! Especially if your process writes lots to stdout/stderr that python 
>> needs to collect.
>>
>> BTW: never ever ever run ANY external process from the webserver: you 
>> could easily be DDoSed AND you incur in lots of issues (random timeouts, 
>> memory issues, leaks, and aforementioned slowness).
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Processing taking a long time in web2py?

[Niphlod]

well... you're asking why opening a process in python rather than doing it 
directly takes more time. no wonders there: of course it takes more time! 
Especially if your process writes lots to stdout/stderr that python needs 
to collect.

BTW: never ever ever run ANY external process from the webserver: you could 
easily be DDoSed AND you incur in lots of issues (random timeouts, memory 
issues, leaks, and aforementioned slowness).

On Thursday, November 17, 2016 at 1:55:28 PM UTC+1, Simon Andersson wrote:
>
> def run():
> import os
> import subprocess
> ex = os.path.join('/home/user', "executable")
>
>
> for i in range(2500):
> a = 1.
> b = 2.
> proc = subprocess.Popen([ex, str(a), str(b),],stdout=subprocess.
> PIPE)
> (out, err) = proc.communicate()
>
>
> redirect(URL('here', args=request.args(0)))
>
> Thanks for the reply. I've written the basic function down. It runs an 
> executable 2500 times.
>
> If I run this in a script it takes 5seconds, and if I run this in web2py 
> it takes 23 seconds.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Script timed out before returning headers for apache server when scheduler task is running

[Niphlod]

are you running sqlite ? if so, AT LEAST run the scheduler on a different 
sqlite database, as sqlite doesn't really withstand lots of concurrent 
writes (that the scheduler does). if your web app shares the same sqlite db 
as the scheduler, it gets blocked whenever the scheduler writes something 
to the db (which happens a LOT!)

On Thursday, November 17, 2016 at 1:25:50 AM UTC+1, Pengfei Yu wrote:
>
> Hi,
>
> I am using web2py as framework for my web application and using the 
> built-in scheduler for the user tasks. They are working fine for more than 
> one years. 
> During the recent several days, I found that whenever I start a task to 
> scheduler, the apache web server is not working, it is stuck in somewhere 
> and give 500 internal server error. If I kill the task process or wait 
> until the task completed, then the apache server goes back to normal. I 
> checked the error log file of apache server, and the error message is 
> "Script timed out before returning headers: wsgihandler.py".
> I think the scheduler task run is independent of the apache web server and 
> just run in background in parallel. I don't know what could be the reason 
> for this problem. The only thing I changed for the scheduler is the change 
> of migrate from True to False when I initiate the scheduler:
> scheduler = Scheduler(db,dict(run=run),migrate=False)
>
> Did anyone encounter similar problem before? I will be very appreciated if 
> someone can provide me a hint to solve this problem.
>
> Thanks very much!
>   
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: what is in the cache ?

[Niphlod]

read CAREFULLY http://web2py.com/books/default/chapter/29/04/the-core#cache 
.

Unfortunately (?) web2py's cache isn't "the usual" cache object, as it's 
designed to store indefinitely values that do not expire automatically. 
Basically time_expire vouches to either fetch the stored value or to update 
the one already in cache.

time_expire=None forces the retrieval of the stored value, without updating 
it, so you MAY use it for your usecase.

NB: The redis backend needs to adhere to the same API web2py has, but 
leverages redis evictions postponing a bit the real expiration.


On Wednesday, November 16, 2016 at 2:53:52 PM UTC+1, Pierre wrote:
>
> how do I retrieve a key,value pair stored in the redis cache ?
>
> am doing this :
>
> message = cache.redis('message', lambda: 'Hello', time_expire=360)
>
>
> now i want to test that key='message' still exists and get its value ?
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: web2py as Grafana endpoint

[Niphlod]

if the request comes in either as a POST or as a PUT it has a body (since 
all examples report a faily nested query structure, I'd say it's the case 
vs a simple querystring in a GET)

If the body carries the correct content-encoding, it's automatically parsed 
into request.vars. You can still read the raw body as usual in request.body 
.



On Tuesday, November 15, 2016 at 10:50:23 PM UTC+1, Dave S wrote:
>
> I'm trying to implement a simple json endpoint for Grafana.  Does anyone 
> here have experience with doing this?
>
>  https://github.com/grafana/simple-json-datasource/blob/master/README.md>
>
> I'm looking at using @service.json, but I'm not sure how to get the body 
> on a /query.
>
> (The sample fake-simple-json-datasource is a server written in 
> javascript!  It doesn't make clear if the /query is a POST or a GET, and if 
> it's a GET how the body gets done.)
>
> /dps
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Starting scheduler worker from controller

[Niphlod]

I'll say it once again: you need a scheduler to run out-of-band pieces of 
things (tasks) that ARE NOT embedded in the web-serving process(es).
At that point you may as well process out-of-band tasks with a simple ajax 
call, and save yourself setting up the scheduler at all.

It's strongly discouraged to start ANY process inside the webserver 
(stderr/stdout clumsiness with wsgi, enforced timeout on the webserver, 
DDoS attacks, etc). Starting a python process inside a controller (even if 
it's the scheduler) has the same exact cons.

The scheduler NEEDS (read: there is absolutely no valid reason not to) to 
be started as an external and independant process from the webserver: if 
you need it to be started when the server comes up,  you NEED to use your 
system's "service" architecture (init.d, systemd, supervisord, windows 
service, etc) .
Or, if you're using uwsgi, you can use its facility to spawn external 
processes (it's in the default installation script).


-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: auth_hasmembership(["admin","supervisor","auditor"]) ?

[Niphlod]

and yes, using a lambda makes that piece of code execute only when that 
decorated function is called rather than every time the controller is hit.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: auth_hasmembership(["admin","supervisor","auditor"]) ?

[Niphlod]

every time you invoke has_membership you ask web2py to verify if in that 
instance the user belongs to a particular group. It translates to 1 or more 
queries to the backend.
"my" method instead looks into the groups stored in the session (fetched 
only once at login), so it doesn't need the roundtrip to check the 
membership.

The only con is that if a user is logged in it retains the membership it 
had at the time of the login, so removal of membership is not "translated" 
to the impossibility to access the decorated function until the user logs 
out and logins again. 

I prefer using relatively short session timeouts and save zillions of db 
queries rather than using has_membership, even if has_membership is 
"strictly" correct (instant verification) while mine isn't.

On Friday, November 11, 2016 at 4:11:46 PM UTC+1, Richard wrote:
>
> Simone,
>
> I don't understand why you say you cached groups, from what you showed you 
> only prevent accessing db until the very end with the lambda no?
>
> Richard
>
> On Fri, Nov 11, 2016 at 8:16 AM, Niphlod <nip...@gmail.com > 
> wrote:
>
>> any and all are definitely python syntax.
>>
>> I do this using the cached groups, so I don't reach the db for every check
>>
>>
>> ALLOWED = set(('admin', 'contributor'))
>> @auth.requires(lambda: auth.user_id and ALLOWED & set(auth.user_groups.
>> values()))
>>
>>
>>
>> On Friday, November 11, 2016 at 1:55:25 PM UTC+1, 黄祥 wrote:
>>>
>>> i'm interested in *ANY* and *ALL* in your code, but can't find anything 
>>> that contain *ANY* or *ALL* on the link you provide, is it DAL, web2py 
>>> or python syntax?
>>>
>>> best regards,
>>> stifan
>>>
>> -- 
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to web2py+un...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: auth_hasmembership(["admin","supervisor","auditor"]) ?

[Niphlod]

any and all are definitely python syntax.

I do this using the cached groups, so I don't reach the db for every check


ALLOWED = set(('admin', 'contributor'))
@auth.requires(lambda: auth.user_id and ALLOWED & set(auth.user_groups.
values()))



On Friday, November 11, 2016 at 1:55:25 PM UTC+1, 黄祥 wrote:
>
> i'm interested in *ANY* and *ALL* in your code, but can't find anything 
> that contain *ANY* or *ALL* on the link you provide, is it DAL, web2py or 
> python syntax?
>
> best regards,
> stifan
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Invalid field name although rname is set

[Niphlod]

update() is a methodyou can't use a model that has a field that 
is named "update". It's true that you're using rname, but the problem is 
that you need to choose a less "funny" name for the field name, not the 
rname.

On Friday, November 11, 2016 at 7:40:32 AM UTC+1, Edwin Haver wrote:
>
>
> Hello,
>
> Why do I get the following error message when I have set the rname for 
> that field and have set check_reserved to None?
>
>  Field: invalid field name: update, use 
> rname for "funny" namesVersion
> web2py™ Version 2.14.6-stable+timestamp.2016.05.10.00.21.47
> Python Python 2.7.12: C:\Python27\python.exe (prefix: C:\Python27)
> Traceback
>
> 1.
> 2.
> 3.
> 4.
> 5.
> 6.
> 7.
> 8.
> 9.
>
> Traceback (most recent call last):
>   File "C:\Users\Edwin\Applications\web2py\gluon\restricted.py", line 227, in 
> restricted
> exec ccode in environment
>   File 
> "C:/Users/Edwin/Applications/web2py/applications/create/models/db_tables.py" 
> , line 
> 24, in 
> Field('update', rname='wac_update', type='string', default=None),
>   File 
> "C:\Users\Edwin\Applications\web2py\gluon\packages\dal\pydal\objects.py", 
> line 1438, in __init__
> 'use rname for "funny" names' % fieldname)
> SyntaxError: Field: invalid field name: update, use rname for "funny" names
>
>
>
> Regards, Edwin
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: strange behavior of $@ bash parameter expansion

[Niphlod]

the error you've seen has nothing to do with either $@ or "$@". it just 
signalled that you had a migration error. once you deleted both metadata 
and the underlying tables, everything was fixed.

On Monday, November 7, 2016 at 6:28:22 PM UTC+1, Mirek Zvolský wrote:
>
> I use Debian (testing), Postgres 9.4,
> and Web2py 2.14.6 (with current trunk is problem the same)
>
> I use such command (assigned to an alias) for long time to start my 
> application
> python web2py.py -K codex2020 -X $@
>
> Today I received strange behaviour, errors like
> web2py.scheduler TICKER: error assigning tasks
> or such something (with no effect if I have deleted scheduler_* tables and 
> their databases/ metadata earlier)
> starting scheduler for "codex2020"... 
> Currently running 1 scheduler processes 
> Traceback (most recent call last): 
>  File "/home/mirek/mz/web2py/gluon/restricted.py", line 227, in restricted 
>exec ccode in environment 
>  File "applications/codex2020/models/scheduler.py", line 80, in  
>scheduler = Scheduler(db) 
>  File "/home/mirek/mz/web2py/gluon/scheduler.py", line 567, in __init__ 
>self.define_tables(db, migrate=migrate) 
>  File "/home/mirek/mz/web2py/gluon/scheduler.py", line 659, in 
> define_tables 
>migrate=self.__get_migrate('scheduler_worker', migrate) 
>  File "/home/mirek/mz/web2py/gluon/packages/dal/pydal/base.py", line 834, 
> in define_table 
>table = self.lazy_define_table(tablename,*fields,**args) 
>  File "/home/mirek/mz/web2py/gluon/packages/dal/pydal/base.py", line 873, 
> in lazy_define_table 
>polymodel=polymodel) 
>  File "/home/mirek/mz/web2py/gluon/packages/dal/pydal/adapters/base.py", 
> line 510, in create_table 
>fake_migrate=fake_migrate 
>  File "/home/mirek/mz/web2py/gluon/packages/dal/pydal/adapters/base.py", 
> line 623, in migrate_table 
>self.execute(sub_query) 
>  File 
> "/home/mirek/mz/web2py/gluon/packages/dal/pydal/adapters/postgres.py", line 
> 362, in execute 
>return BaseAdapter.execute(self, *a, **b) 
>  File "/home/mirek/mz/web2py/gluon/packages/dal/pydal/adapters/base.py", 
> line 1388, in execute 
>return self.log_execute(*a, **b) 
>  File "/home/mirek/mz/web2py/gluon/packages/dal/pydal/adapters/base.py", 
> line 1382, in log_execute 
>ret = self.get_cursor().execute(command, *a[1:], **b) 
> ProgrammingError: column "worker_stats__tmp" is of type json but 
> expression is of type text 
> LINE 1: UPDATE scheduler_worker SET worker_stats__tmp=worker_stats; 
>  ^ 
> HINT:  You will need to rewrite or cast the expression.
>
> Then I decided then if I run without $@, all will start well:
> python web2py.py -K codex2020 -X
>
> After testing of everything I went back to:
> python web2py.py -K codex2020 -X $@
> or
> python web2py.py -K codex2020 -X "$@"
>
> and I have now no more the previous problem. All works well.
>
> Has somebody any idea about the reason of problems
> and can you explain me the difference between $@ and "$@". Which one is 
> more safe to use? Is there a difference?
>
> Thanks...
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: scheduler creates "zombie" processes

[Niphlod]

there was an issue on more than 1kb that showed pretty much a limit on 
python's implementation, that's solved in recent releases of the scheduler 
but I wonder if the "tax" on that sem_wait() is trying to flush off or read 
the buffered output of the task.

On Friday, November 4, 2016 at 11:50:14 PM UTC+1, Erwn Ltmann wrote:
>
> On Friday, November 4, 2016 at 10:44:31 PM UTC+1, Niphlod wrote:
>>
>> BTW: do your task write a lot on stdout/stderr and/or return huge results 
>> ?
>>
>
> Probably. I wrote into the log more than usual. :( Mostly DEBUGs - it's 
> possible to switch off them.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Making two queries into one

[Niphlod]

it seems that on the whole set there's something missing

1st query
indiban
anos
empresas
*indicadores*

2nd query
eleban
anos
empresas
*elementos*

moreover, in your "DAL syntax", only the references to those external 
tables are taken.
The real missing link that could get you the answer is how to link 
elementos and indicadores (or, more specifically, indicadores_id and 
elementos_id). Without that, there's - apparently - no way to link eleban 
and indiban. 

On Saturday, November 5, 2016 at 3:16:10 PM UTC+1, Gualter Portella wrote:
>
> Dear all,
>
> I hope this e-mail finds you all well.
>
> I have written two queries to retrieve data from my database (model). I 
> did them initially through plain SQL, then I followed the DAL syntax. They 
> all return the same output regardless the syntax chosen. Please, see the 
> code below:
>
> Two different controllers:
>
> def sql():
> sql = "SELECT indiban.valor, meses.abrev, anos.numero, empresas.nome "
> sql = sql + "FROM indiban INNER JOIN meses ON indiban.meses_id = 
> meses.id "
> sql = sql + "INNER JOIN anos ON indiban.anos_id = anos.id "
> sql = sql + "INNER JOIN indicadores ON indiban.indicadores_id = 
> indicadores.id "
> sql = sql + "INNER JOIN empresas ON indiban.empresas_id = empresas.id 
> "
> sql = sql + "WHERE indicadores.id = 3 "
> sql = sql + "AND meses.id = 5 AND anos.id = 22 Limit 10;"
> ratios = db.executesql(sql)
> 
> sql = "SELECT eleban.valor, meses.abrev, anos.numero, empresas.nome "
> sql = sql + "FROM eleban INNER JOIN meses ON eleban.meses_id = 
> meses.id "
> sql = sql + "INNER JOIN anos ON eleban.anos_id = anos.id "
> sql = sql + "INNER JOIN elementos ON eleban.elementos_id = 
> elementos.id  "
> sql = sql + "INNER JOIN empresas ON eleban.empresas_id = empresas.id "
> sql = sql + "WHERE elementos.id = 5 "
> sql = sql + "AND meses.id = 5 AND anos.id = 22 Limit 10;"
> accounts = db.executesql(sql)
> 
> return locals()
>
> def sfn():
>
>ratios = db((Indiban.meses_id==mes_id)&(Indiban.anos_id==ano_id)&
> 
> (Indiban.indicadores_id==rspla_id)).select(Indiban.empresas_id, 
> Indiban.valor,
>
> Indiban.meses_id, Indiban.anos_id, limitby=(0, 10))
>
>accounts = db((Eleban.meses_id==mes_id)&(Eleban.anos_id==ano_id)&
> 
> (Eleban.elementos_id==ativo_id)).select(Eleban.empresas_id, Eleban.valor,
>
> Eleban.meses_id, Eleban.anos_id, limitby=(0, 10))
>
>
> return locals()
>
> DAL output:
>
> indiban.empresas_idindiban.
> valorindiban.meses_idindiban.anos_idSISTEMA FINANCEIRO 
> NACIONAL0.1300052016BANCO DO BRASIL0.1100052016BRB - BANCO DE 
> BRASILIA0.0200052016POTTENCIAL-0.0100052016CAIXA ECONOMICA 
> FEDERAL0.0300052016BANCO INTERMEDIUM0.0400052016RIBEIRAO 
> PRETO0.1200052016BGN 
> CETELEM-0.1200052016SEMEAR-0.0100052016BM 
> --
>
> eleban.empresas_ideleban.valoreleban.meses_ideleban.anos_idSISTEMA 
> FINANCEIRO NACIONAL7404931306106.43052016BANCO DO 
> BRASIL1070715292094.95052016BRB - BANCO DE 
> BRASILIA13013438306.52052016POTTENCIAL167724699.00052016CAIXA ECONOMICA 
> FEDERAL1096578802720.18052016BANCO INTERMEDIUM2745032207.17052016RIBEIRAO 
> PRETO486037290.78052016BGN 
> CETELEM8371996257.63052016SEMEAR717457527.11052016BM
>  
>
> SQL Output: 
>
> ((Decimal('0.1300'), u'mai', u'2016', u'SISTEMA FINANCEIRO NACIONAL'), 
> (Decimal('0.1100'), u'mai', u'2016', u'BANCO DO BRASIL'), 
> (Decimal('0.0200'), u'mai', u'2016', u'BRB - BANCO DE BRASILIA'), 
> (Decimal('-0.0100'), u'mai', u'2016', u'POTTENCIAL'), (Decimal('0.0300'), 
> u'mai', u'2016', u'CAIXA ECONOMICA FEDERAL'), (Decimal('0.0400'), u'mai', 
> u'2016', u'BANCO INTERMEDIUM'), (Decimal('0.1200'), u'mai', u'2016', 
> u'RIBEIRAO PRETO'), (Decimal('-0.1200'), u'mai', u'2016', u'BGN CETELEM'), 
> (Decimal('-0.0100'), u'mai', u'2016', u'SEMEAR'), (Decimal('0.1500'), 
> u'mai', u'2016', u'BM')) 
> --
> ((Decimal('7404931306106.43'), u'mai', u'2016', u'SISTEMA FINANCEIRO 
> NACIONAL'), (Decimal('1070715292094.95'), u'mai', u'2016', u'BANCO DO 
> BRASIL'), (Decimal('13013438306.52'), u'mai', u'2016', u'BRB - BANCO DE 
> BRASILIA'), (Decimal('167724699.00'), u'mai', u'2016', u'POTTENCIAL'), 
> (Decimal('1096578802720.18'), u'mai', u'2016', u'CAIXA ECONOMICA FEDERAL'), 
> (Decimal('2745032207.17'), u'mai', u'2016', u'BANCO INTERMEDIUM'), 
> (Decimal('486037290.78'), u'mai', u'2016', u'RIBEIRAO PRETO'), 
> (Decimal('8371996257.63'), u'mai', u'2016', u'BGN CETELEM'), 
> (Decimal('717457527.11'), u'mai', u'2016', u'SEMEAR'), 
> (Decimal('234811696.13'), u'mai', u'2016', u'BM')) 
>
> The data  in the output are identical as expected. What I want to do is  
> to combine  the two queries into one, preferably in DAL, so that I do not 
> need to use two loops in order to insert the indiban.valor and eleban.valor 

Re: [web2py] Re: switch to python 3

[Niphlod]

the same gain moving any piece of code from python 2 to python 3 (read on 
the interwebs, there are a few).
at least now you can choose and have web2py not standing in the way.

On Sunday, November 6, 2016 at 12:34:39 AM UTC+1, Ramos wrote:
>
> what is the gain moving to python 3?
>
> 2016-11-05 17:59 GMT+00:00 Leonel Câmara  >:
>
>> web2py is almost completely python 3 compatible. You can follow the issue 
>> here: https://github.com/web2py/web2py/issues/1353
>>
>> -- 
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to web2py+un...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: scheduler creates "zombie" processes

[Niphlod]

BTW: do your task write a lot on stdout/stderr and/or return huge results ?

On Friday, November 4, 2016 at 10:43:34 PM UTC+1, Niphlod wrote:
>
> well, on unix python docs report that p.terminate() use sigkill, so I 
> wonder why in your env one works and not the other...
>
> On Friday, November 4, 2016 at 8:57:34 AM UTC+1, Erwn Ltmann wrote:
>>
>> BTW: if I use os.kill(p.pid,signal.SIGKILL) instead of p.termiate() 
>> everithing is fine.
>>
>> 11-04 08:40:59.521 [13556] web2py.Scheduler - DEBUG -  MD: terminating 
>>> (os kill)
>>> 11-04 08:40:59.521 [13556] web2py.Scheduler - DEBUG -  MD: terminated
>>> 11-04 08:40:59.522 [13556] web2py.Scheduler - DEBUG -  MD: joining
>>> 11-04 08:40:59.524 [13556] web2py.Scheduler - DEBUG -  MD: joined
>>>
>>
>> No running and still waiting processes are left. :)
>>
>> But, if I configure timeouts longer then 30 seconds - maybe 10 minutes or 
>> more, then the worker is still waiting until this timeout. Because of that 
>> I have to find the reason for sem_wait().
>>
>> Erwn
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: scheduler creates "zombie" processes

[Niphlod]

well, on unix python docs report that p.terminate() use sigkill, so I 
wonder why in your env one works and not the other...

On Friday, November 4, 2016 at 8:57:34 AM UTC+1, Erwn Ltmann wrote:
>
> BTW: if I use os.kill(p.pid,signal.SIGKILL) instead of p.termiate() 
> everithing is fine.
>
> 11-04 08:40:59.521 [13556] web2py.Scheduler - DEBUG -  MD: terminating (os 
>> kill)
>> 11-04 08:40:59.521 [13556] web2py.Scheduler - DEBUG -  MD: terminated
>> 11-04 08:40:59.522 [13556] web2py.Scheduler - DEBUG -  MD: joining
>> 11-04 08:40:59.524 [13556] web2py.Scheduler - DEBUG -  MD: joined
>>
>
> No running and still waiting processes are left. :)
>
> But, if I configure timeouts longer then 30 seconds - maybe 10 minutes or 
> more, then the worker is still waiting until this timeout. Because of that 
> I have to find the reason for sem_wait().
>
> Erwn
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: scheduler creates "zombie" processes

[Niphlod]

Got that. Thanks for explaining your POV. What I was trying to say is that 
without being able to reproduce, I can only guess. And since the scheduler 
code in regards of process handling is quite streamlined, there are not a 
lot of places where a zombie can be created.
So, let's guess...

Are zombie processes "still running and churning data" or just leftovers ? 
Usually there are two kinds of zombie processes: completely detached 
processes running and consuming resources (what theoretically the scheduler 
should avoid at all costs) and simple leftovers on the "virtual" process 
table that are there because nobody (the parent process, usually) reclaimed 
either their output or their status code. The second is something to avoid, 
of course, but shouldn't hurt the functionality of the app.

from the pstree you posted, it seems that (shorting code here and there)

p = multiprocessing.Process(executor, )
p.start()
try:
#task runs
p.join(run_timeout)
except:
#this should be raised only when a general error on the task happened, 
so it's a STOPPED one
p.terminate()
p.join()
else:
#this is the codepath your task takes, since its the one landing 
TIMEOUT tasks
if p.is_alive():
 # this is ultimately the call that SHOULD kill the process you 
later find as a zombie
 p.terminate()
 

this "terminate" is labelled as using SIGKILL on the process. Eventual 
processes child of that one (i.e. a subprocess call inside the task itself) 
are not guaranteed to be terminated, but then they'll show as orphaned, 
while your pstree reports python processes still "attached" to the 
scheduler worker process. 

>From where I stand, if the result is the task being labelled as TIMEOUT 
(with the corresponding "task timeout" debug line), it can only be 
originated there.
Maybe there's a culprit there can you add a p.join() after that 
p.terminate(), and maybe a few debug lines ? 
i.e.


... else:
if p.is_alive():
logger.debug(' MD: terminating')
p.terminate()
logger.debug(' MD: terminated')
logger.debug(' MD: joining')
p.join()
logger.debug(' MD: joined')
logger.debug(' task timeout')
try:
# we try to get a traceback here
tr = queue.get(timeout=2)
...




-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: web2py for windows server. IIS or Nginx?

[Niphlod]

IIS and ISAPI --> helicon ? AFAIK it was the "old" standard.
When Azure eventually moved to support python web apps, smart guys at MS 
implemented the - until that moment - missing link to run natively a 
fastcgi python process speaking with IIS through the "same interface" that 
runs any other fastcgi-enabled language (e.g. PHP), which is the 
wfastcgi.py module/script. 
I'd go for the "newly and improved" rather than having a 3rd party 
integrated with ISAPI: less moving parts, better support moving forward as 
it's "100%" MS supported code.

On Thursday, November 3, 2016 at 12:45:36 AM UTC+1, Pbop wrote:
>
> We built a SHIB SSO and OAUTH SSO middleware in web2py that handles 1000+ 
> concurrent users with very good results in IIS and ISAPI running a WSGI 
> wrapper ala the cookbook instructions. I hear IIS and FastCGI and wonder if 
> this is a better deployment option? Are you aware of any advantages of one 
> over the other?
>
> On Wednesday, November 2, 2016 at 5:55:36 PM UTC-4, Niphlod wrote:
>>
>> it's from some time ago but should work without issues. I think some 
>> improvements have been made regarding wfastcgi availability as a proper 
>> package, but in any case, I'm here to help (and maybe we can revisit the 
>> segment on the book with your experience).
>>
>> <tl;dr>: should be "verbatim" but improvements on MS side could have 
>> changed little bits. 
>>
>> On Wednesday, November 2, 2016 at 2:09:07 PM UTC+1, Omi Chiba wrote:
>>>
>>> OK! I failed last time but it worth try again. I will follow the book 
>>> try it later.
>>>
>>> http://www.web2py.com/books/default/chapter/29/13/deployment-recipes?search=iis#IIS
>>>
>>> On Wednesday, November 2, 2016 at 3:43:37 AM UTC-5, Niphlod wrote:
>>>>
>>>> *nix --> nginx with uwsgi
>>>> windows --> iis with fastcgi
>>>>
>>>>
>>>> BTW: iis is perfectly fine running production envs. And it's NOT 
>>>> difficult.
>>>>
>>>>
>>>> On Tuesday, November 1, 2016 at 8:01:22 PM UTC+1, Omi Chiba wrote:
>>>>>
>>>>> Thank you! I will stick to Apache/wsgi for now. 
>>>>>
>>>>> On Tuesday, November 1, 2016 at 1:54:37 PM UTC-5, Jim S wrote:
>>>>>>
>>>>>> I think the thread you referenced was one discussing deployment on 
>>>>>> Ubuntu where yes, nginx/uwsgi is preferred.  But, the Windows 
>>>>>> environment 
>>>>>> is different (in my opinion) since the Windows nginx is still considered 
>>>>>> 'beta'.   I wouldn't feel that comfortable using it.  (that said, I do 
>>>>>> use 
>>>>>> many other 'beta' projects)
>>>>>>
>>>>>> If Apache/mod_wsgi is really frowned upon, should it be noted in the 
>>>>>> book?
>>>>>>
>>>>>> @omi - I migrated from Windows/Apache/mod_wsgi a while back to 
>>>>>> Ubuntu/nginx/uwsgi.  I think that at that time I switched from using 
>>>>>> pyodbc 
>>>>>> to pypyodbc.  I too access MS SQL servers from my ubuntu box using ODBC 
>>>>>> (along with IBM AS/400 databases).  It works very well for me.
>>>>>>
>>>>>> -Jim
>>>>>>
>>>>>>
>>>>>> On Tuesday, November 1, 2016 at 1:33:53 PM UTC-5, Richard wrote:
>>>>>>>
>>>>>>> I would say, we don't not don't support it, we just don't maintain a 
>>>>>>> web2py setup script with Apache... I think the decision was to reduce 
>>>>>>> the 
>>>>>>> number of setup script to the bare minium to only the one web2py-devs 
>>>>>>> are 
>>>>>>> willing to maintain...
>>>>>>>
>>>>>>> That been said, I am sure that if you or someone else take owner 
>>>>>>> ship to update and maintain Apache setup script because it important 
>>>>>>> for 
>>>>>>> you we will bring it back in the scripts folder... But I wouldn't take 
>>>>>>> that 
>>>>>>> path before someone demonstrate commitment to the task as we don't want 
>>>>>>> to 
>>>>>>> get back stuff that will not be maintain in years in the repo... I 
>>>>>>> guess 
>>>>>>> you can set your own github repo to demonstrate you

Re: [web2py] .git file in gluon/packages/dal/.git

[Niphlod]

ok, now it's clear. probably a packaging error on mdipierro's side.

On Thursday, November 3, 2016 at 2:48:45 PM UTC+1, Giles Thomas wrote:
>
> @Niphlod -- the confusing thing for me is that the .git file is present in 
> the source download from the web2py website.  It makes perfect sense that 
> it's present when you do a git checkout, but not when you've downloaded a 
> zip file.
>
>
> All the best,
>
> Giles
>
>
> On Wednesday, 2 November 2016 11:02:18 UTC, Niphlod wrote:
>>
>> uhm ... what ?
>>
>> On Tuesday, November 1, 2016 at 7:10:16 PM UTC+1, Richard wrote:
>>>
>>> I agree with you Simone, but should it stay there in the build for pydal 
>>> only?
>>>
>>> Richard
>>>
>>> On Tue, Nov 1, 2016 at 1:45 PM, Niphlod <nip...@gmail.com> wrote:
>>>
>>>> it's perfectly normal as pydal is a subrepo (it's in the readme, too)
>>>>
>>>>
>>>> On Tuesday, November 1, 2016 at 5:43:44 PM UTC+1, Giles Thomas wrote:
>>>>>
>>>>> Thanks, Richard.  
>>>>>
>>>>> Regarding why -- I'm talking about the version that will be installed 
>>>>> by default when someone creates a web2py app on the hosting platform my 
>>>>> company provides, PythonAnywhere -- which would also affect the version 
>>>>> used by the "Try it now online" link on the front page of 
>>>>> www.web2py.com.  I agree that doing version control the web2py way is 
>>>>> probably more sensible than making the whole web2py folder a git repo, 
>>>>> but 
>>>>> if people do want to follow the git route, it would be better if we 
>>>>> didn't 
>>>>> have that "trap" in there for them :-)
>>>>>
>>>>>
>>>>> All the best,
>>>>>
>>>>> Giles
>>>>>
>>>>>
>>>>> On Tuesday, 1 November 2016 16:13:22 UTC, Richard wrote:
>>>>>>
>>>>>> Sorry, you right in there is gluon/packages/dal/.git, I thought you 
>>>>>> were speaking of a .git/ folder at the root of web2py which there is 
>>>>>> not...
>>>>>>
>>>>>> I guess you can delete it if it cause problem, but if you have issue 
>>>>>> with it, it means you try to init you git repo over the entire web2py 
>>>>>> folder?
>>>>>>
>>>>>> Why don't you just version control your app? which is what we usually 
>>>>>> do...
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Richard
>>>>>>
>>>>>> On Tue, Nov 1, 2016 at 10:51 AM, Giles Thomas <giles@gmail.com> 
>>>>>> wrote:
>>>>>>
>>>>>>> The one labelled "For normal users", so presumably stable.  The 
>>>>>>> VERSION file says:
>>>>>>>
>>>>>>> Version 2.14.6-stable+timestamp.2016.05.10.00.21.47
>>>>>>>
>>>>>>>
>>>>>>> All the best,
>>>>>>>
>>>>>>> Giles
>>>>>>>
>>>>>>> On Tuesday, 1 November 2016 14:48:03 UTC, Richard wrote:
>>>>>>>>
>>>>>>>> Nightly or stable? I don't have it in my last build 2.14.6 I 
>>>>>>>> think... 
>>>>>>>>
>>>>>>>> Richard
>>>>>>>>
>>>>>>>> On Tue, Nov 1, 2016 at 10:42 AM, Giles Thomas <giles@gmail.com> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This is in a download from web2py.com -- specifically, the 
>>>>>>>>> "Source code" download from 
>>>>>>>>> http://www.web2py.com/init/default/download.
>>>>>>>>>
>>>>>>>>> Interestingly, gluon/packages/dal/.git is a file rather than a 
>>>>>>>>> directory.  It contains this:
>>>>>>>>>
>>>>>>>>> gitdir: ../../../.git/modules/gluon/packages/dal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> All the best,
>>>>>>>>>
>>>>>>>>> Giles
>>>>>&g

Re: [web2py] Re: web2py for windows server. IIS or Nginx?

[Niphlod]

it's from some time ago but should work without issues. I think some 
improvements have been made regarding wfastcgi availability as a proper 
package, but in any case, I'm here to help (and maybe we can revisit the 
segment on the book with your experience).

<tl;dr>: should be "verbatim" but improvements on MS side could have 
changed little bits. 

On Wednesday, November 2, 2016 at 2:09:07 PM UTC+1, Omi Chiba wrote:
>
> OK! I failed last time but it worth try again. I will follow the book try 
> it later.
>
> http://www.web2py.com/books/default/chapter/29/13/deployment-recipes?search=iis#IIS
>
> On Wednesday, November 2, 2016 at 3:43:37 AM UTC-5, Niphlod wrote:
>>
>> *nix --> nginx with uwsgi
>> windows --> iis with fastcgi
>>
>>
>> BTW: iis is perfectly fine running production envs. And it's NOT 
>> difficult.
>>
>>
>> On Tuesday, November 1, 2016 at 8:01:22 PM UTC+1, Omi Chiba wrote:
>>>
>>> Thank you! I will stick to Apache/wsgi for now. 
>>>
>>> On Tuesday, November 1, 2016 at 1:54:37 PM UTC-5, Jim S wrote:
>>>>
>>>> I think the thread you referenced was one discussing deployment on 
>>>> Ubuntu where yes, nginx/uwsgi is preferred.  But, the Windows environment 
>>>> is different (in my opinion) since the Windows nginx is still considered 
>>>> 'beta'.   I wouldn't feel that comfortable using it.  (that said, I do use 
>>>> many other 'beta' projects)
>>>>
>>>> If Apache/mod_wsgi is really frowned upon, should it be noted in the 
>>>> book?
>>>>
>>>> @omi - I migrated from Windows/Apache/mod_wsgi a while back to 
>>>> Ubuntu/nginx/uwsgi.  I think that at that time I switched from using 
>>>> pyodbc 
>>>> to pypyodbc.  I too access MS SQL servers from my ubuntu box using ODBC 
>>>> (along with IBM AS/400 databases).  It works very well for me.
>>>>
>>>> -Jim
>>>>
>>>>
>>>> On Tuesday, November 1, 2016 at 1:33:53 PM UTC-5, Richard wrote:
>>>>>
>>>>> I would say, we don't not don't support it, we just don't maintain a 
>>>>> web2py setup script with Apache... I think the decision was to reduce the 
>>>>> number of setup script to the bare minium to only the one web2py-devs are 
>>>>> willing to maintain...
>>>>>
>>>>> That been said, I am sure that if you or someone else take owner ship 
>>>>> to update and maintain Apache setup script because it important for you 
>>>>> we 
>>>>> will bring it back in the scripts folder... But I wouldn't take that path 
>>>>> before someone demonstrate commitment to the task as we don't want to get 
>>>>> back stuff that will not be maintain in years in the repo... I guess you 
>>>>> can set your own github repo to demonstrate your commitment and help the 
>>>>> community though, and it could be reference somewhere appropriate in the 
>>>>> book.
>>>>>
>>>>> Richard
>>>>>
>>>>> On Tue, Nov 1, 2016 at 2:22 PM, Dave S <snide...@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tuesday, November 1, 2016 at 7:51:26 AM UTC-7, Omi Chiba wrote:
>>>>>>>
>>>>>>> Thank you. I thought the Massimo's comment below and he  also 
>>>>>>> mentioned somewhere we don't want to support Apache anymore... that's 
>>>>>>> why I 
>>>>>>> was nervous. I was thinking to your direction (Moving to Ubuntu) but I 
>>>>>>> use 
>>>>>>> pyodbc to connect Microsoft SQL Server and DB2, also python-ldap.. so 
>>>>>>> not 
>>>>>>> sure if it works the same way.
>>>>>>>
>>>>>>>
>>>>>>> "P.S. I stand by Niphlod. He did not say anything offending and his 
>>>>>>> comment was insightful. We do not recommend apache+mod_wsgi because 
>>>>>>> there 
>>>>>>> are better ways (nginx+uwsgi)."
>>>>>>>
>>>>>>>
>>>>>> If you have a working Apache configuration, that's an argument for 
>>>>>> staying with it [caveats follow].  Part of the deprecating is because 
>>>>>> Apache configuration is delicate, complicated, and [reportedly] the 
>>>>>> documentation isn'

[web2py] Re: scheduler creates "zombie" processes

[Niphlod]

I'm naturally curious to know the culprit in your env, but unfortunately I 
can't reproduce in any of my systems (*buntu 14.04 + postgresql/sqlite or 
Win2012r2  + postgresql/sqlite/mssql).
Needs to be said that I mostly hate mysql (and would never trust it nor 
recommend to the worst of my enemies) but its totally a personal bias. I 
know several peoples using mysql and never reporting issues whatsoever.

Dunno if I'm the "most powered user" of the scheduler around but I range 
from 4 to 20 workers (even 50, but "cheating" with the redis-backed 
version) and these kind of issues never happened, unless a relly long 
task with a raally long stdout output (which, once known, it's easy to 
suppress, and btw the problem is less of a matter since a few releases 
ago). 
"My" worker processes are usually alive for more than a week (i.e. same 
pid), so it's not a problem of phantom processes or leaks.

Although pretty heavy (resources-wise) if compared to pyres, celery, huey, 
rq (just to name the "famous ones") , web2py's scheduler, which HAS to work 
in all supported OS by web2py is rather simple: each worker spawns a single 
process with a queue 1 element long to communicate over ... that process 
handles a single task, and then dies. Everything is thrashed and recreated 
at the next task pick-up.



On Wednesday, November 2, 2016 at 5:47:42 PM UTC+1, Erwn Ltmann wrote:
>
> Hi Niphlod,
>
> your replies are always a pleasure to me. :)
>
> On Wednesday, November 2, 2016 at 12:00:48 PM UTC+1, Niphlod wrote:
>>
>> I'd say there are a LOT of strange things going on on your system, since 
>> you're reporting several different issues that nobody ever faced and all in 
>> the last week.
>>
>
> Concerning deadlocks and zombies - right? Both issues are faced within the 
> using the scheduler, not web2py in general. And only in cases I start more 
> than one worker.
>
> zombie processes shouldn't be there unless you killed improperly a worker 
>> process. 
>> Python can't really do anything about it, and that's the way there's a 
>> specific API to kill (or terminate) a worker.
>>
>
> Your right, the killer is the scheduler himself. Why? The scheduler 
> terminates a task after passing the timeout. The timeout happened because 
> the task never does that as it is defined. In cases of zombie situation the 
> sub process is stopping with sem_wait() function (pstack). I don't know 
> way. But, it's happened before the function 'executor' entered, because of 
> no debug line printing at the entry point of that function.
>
> Ok. That's all what I know. I have different RHEL systems (RH6,RH5) with 
> python 2.7.12 and MariaDB. Not realy exotic.
>
> Thank you for your endurance
> Erwn
>
>  
>
>>
>> On Wednesday, November 2, 2016 at 10:53:58 AM UTC+1, Erwn Ltmann wrote:
>>>
>>> Dear all,
>>>
>>> I'm astonished about a lot of processes as sub process of scheduler 
>>> worker are not finished.
>>>
>>> pstree -p 16731
>>>  
>>>>
>>>> bash(16731)---python2.7(24545)-+-python2.7(24564)---{python2.7}(24565)
>>>>|-python2.7(24572)-+-python2.7(1110)
>>>>|  |-python2.7(8647)
>>>>|  |-python2.7(11747)
>>>>|  |-python2.7(14117)
>>>>|  |-python2.7(14302)
>>>>
>>>
>>> The 16731 is my shell I started the scheduler with four worker:
>>>
>>> w2p -K arm:ticker,arm,arm,arm
>>>>
>>>
>>> The pid 24564 is the ticker worker (only hold the ticker) and 24572 one 
>>> of three standard worker which has to process my task's function.
>>>
>>> My first focus was on the function itself. But, if I clip the function 
>>> ('return True' at start point) the zombies were already there. My next 
>>> analyze step was to show the pid at the start point of 'executor' function 
>>> of scheduler.py. In case of zombie processes I never reach this debug 
>>> point. Next I printed out the list of zombie processes 
>>> (multiprocessing.active_children()) at the exit point of tasks which passed 
>>> the timeout (see function async). It's the point in the scheduler code 
>>> where 'task timeout' is printing out. The timeout is clear because of a 
>>> process which never returns a result. But, how is it possible?
>>>
>>> Here's the list of my extra debug line in function async's timeout part:
>&g

[web2py] Re: How to start a scheduler task from a module?

[Niphlod]

tl;dr : it just gets automatically filled with 
"request.application/request.controller" in a web environment that it is 
missing in your module.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: How to start a scheduler task from a module?

[Niphlod]

well

mysched.queue_task(application_name='yourappname')


BTW: application_name is usually either "appname" or 
"appname/controller_name" (the latter useful in cases where conditional 
models are required an/or the task is defined in the controller rather than 
in a model)



On Wednesday, November 2, 2016 at 9:58:57 PM UTC+1, mweissen wrote:
>
> Thanks, I have tried it, but the scheduler needs an "application_name":
>
> In the module:
>
> db = DAL()
> scheduler = Scheduler(db, migrate=False)
> rtn = scheduler.queue_task("mytask")
>
>
> Now the return value of queue_task is
> rtn ==  {'errors': {'application_name': 'Enter a value'}, 'id': None, 
> 'uuid': None}
>
> How can I provide an application_name ?
>
> 2016-11-02 11:58 GMT+01:00 Niphlod <nip...@gmail.com >:
>
>> if the scheduler is already defined in the db, just use 
>> Scheduler(the_uri. migrate=False) . there shouldn't be issues with queuing 
>> tasks .
>>
>> On Wednesday, November 2, 2016 at 11:17:00 AM UTC+1, mweissen wrote:
>>>
>>> Thank you, but it did not work, because the module program does not use 
>>> the models.
>>>
>>> More details:
>>>
>>> The module contains a small smtp-server. This program writes incoming 
>>> emails to a table "db.emails".  This part works very well. Now I want to 
>>> start a scheduler action to analyze each email and to do something with 
>>> every email. The scheduler is necessary, because every action with an email 
>>> may consume some time to finish.
>>>
>>>
>>> 2016-11-02 1:05 GMT+01:00 Dave S <snide...@gmail.com>:
>>>
>>>>
>>>>
>>>> On Tuesday, November 1, 2016 at 2:42:28 PM UTC-7, mweissen wrote:
>>>>>
>>>>> I want to start a task from a module. (The module contains a small 
>>>>> smtp-server.)
>>>>> I could it with a db.scheduler_task.insert(...), but I want to use 
>>>>> myschedule.queue_task(...)
>>>>>
>>>>> ​What would be the correct way to define "myschedule" ? Something like
>>>>>
>>>>> myschedule = Scheduler(current.db)​
>>>>>
>>>>>
>>>>> ​Regards, Martin​
>>>>>
>>>>
>>>>
>>>> My first thought is that you define myschedule in your model file(s), 
>>>> and then in whatever controller calls your module, add it to current and 
>>>> pass it that way, or make it an argument of your module's function.  I 
>>>> don't see any reason NOT to use queue_task, and inserting directly into 
>>>> the 
>>>> table is subject Change Without Notice.
>>>>
>>>> /dps
>>>>
>>>>
>>>>
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: DEADLOCKs between two or more scheduler worker

[Niphlod]

ahem  until "it makes sense" any modification (and discussion about it) 
doesn't really help anyone ^_^ . 
The fact that you have 4 workers and a congestion problem gives me the hint 
that your db is on the lower side of the needed specs for a normal server. 
These kind of issues starts to show with 40-50 workers with a disk on an 
usb drive not on a production server (even built 10 years ago).

On Tuesday, November 1, 2016 at 2:42:32 PM UTC+1, Erwn Ltmann wrote:
>
> Hi Niphold,
>
> you are right: I have an extra database select in order to get the list of 
> dead workers.
>
> Usually I have four workers for example. They are static and shouldn't 
> terminate often. In this case, I call only once the database in order to 
> get the list of dead workers and I assume this list is always empty. In 
> this case nothing is to do. The inner part of my condition will be 
> important very rare and because of that I can ignore this within my runtime 
> complexity calculation. In our original code we call always twice the 
> question how many dead workers there are (update and delete). My suggestion 
> reflects a runtime rate of 1 for 2.
>
> Anyway, if I run the worker with my suggested extra condition I could 
> eliminate the deadlock cases. This works very well because the extra 
> condition. I am happy :)
>
> Thx a lot.
> Erwn
>
> On Monday, October 31, 2016 at 3:02:42 PM UTC+1, Niphlod wrote:
>>
>> sorry, but it doesn't really make sense. 
>> You're executing twice the same command (the call enclosed in len() and 
>> the actual .delete() call), which is the counter-arg for relaxing a 
>> pressured database environment. 
>>
>> On Monday, October 31, 2016 at 2:04:24 PM UTC+1, Erwn Ltmann wrote:
>>>
>>> Hi,
>>>
>>> thank you for your reply.
>>>
>>> @Pierre: MariaDB (in my case) handled deadlocks automaticly too. Good to 
>>> known, I don't have to be worry about that.
>>>
>>> @Niphlod: I tried to beef up my database host. No effects. Another 
>>> suggestion is to prevent the cases for such situation. I did it by an 
>>> another extra code line in your worker function send_heartbeat:
>>>
>>> *if len(db.executesql(dead_workers_name)) > 0:*
>>>>   db(
>>>>(st.assigned_worker_name.belongs(dead_workers_name)) &
>>>>(st.status == RUNNING)
>>>>   ).update(assigned_worker_name='', status=QUEUED)
>>>>   dead_workers.delete()
>>>>
>>>
>>>
>>>
>>> Erwn
>>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] .git file in gluon/packages/dal/.git

[Niphlod]

uhm ... what ?

On Tuesday, November 1, 2016 at 7:10:16 PM UTC+1, Richard wrote:
>
> I agree with you Simone, but should it stay there in the build for pydal 
> only?
>
> Richard
>
> On Tue, Nov 1, 2016 at 1:45 PM, Niphlod <nip...@gmail.com > 
> wrote:
>
>> it's perfectly normal as pydal is a subrepo (it's in the readme, too)
>>
>>
>> On Tuesday, November 1, 2016 at 5:43:44 PM UTC+1, Giles Thomas wrote:
>>>
>>> Thanks, Richard.  
>>>
>>> Regarding why -- I'm talking about the version that will be installed by 
>>> default when someone creates a web2py app on the hosting platform my 
>>> company provides, PythonAnywhere -- which would also affect the version 
>>> used by the "Try it now online" link on the front page of www.web2py.com.  
>>> I agree that doing version control the web2py way is probably more sensible 
>>> than making the whole web2py folder a git repo, but if people do want to 
>>> follow the git route, it would be better if we didn't have that "trap" in 
>>> there for them :-)
>>>
>>>
>>> All the best,
>>>
>>> Giles
>>>
>>>
>>> On Tuesday, 1 November 2016 16:13:22 UTC, Richard wrote:
>>>>
>>>> Sorry, you right in there is gluon/packages/dal/.git, I thought you 
>>>> were speaking of a .git/ folder at the root of web2py which there is not...
>>>>
>>>> I guess you can delete it if it cause problem, but if you have issue 
>>>> with it, it means you try to init you git repo over the entire web2py 
>>>> folder?
>>>>
>>>> Why don't you just version control your app? which is what we usually 
>>>> do...
>>>>
>>>> Thanks
>>>>
>>>> Richard
>>>>
>>>> On Tue, Nov 1, 2016 at 10:51 AM, Giles Thomas <giles@gmail.com> 
>>>> wrote:
>>>>
>>>>> The one labelled "For normal users", so presumably stable.  The 
>>>>> VERSION file says:
>>>>>
>>>>> Version 2.14.6-stable+timestamp.2016.05.10.00.21.47
>>>>>
>>>>>
>>>>> All the best,
>>>>>
>>>>> Giles
>>>>>
>>>>> On Tuesday, 1 November 2016 14:48:03 UTC, Richard wrote:
>>>>>>
>>>>>> Nightly or stable? I don't have it in my last build 2.14.6 I think... 
>>>>>>
>>>>>> Richard
>>>>>>
>>>>>> On Tue, Nov 1, 2016 at 10:42 AM, Giles Thomas <giles@gmail.com> 
>>>>>> wrote:
>>>>>>
>>>>>>> This is in a download from web2py.com -- specifically, the "Source 
>>>>>>> code" download from http://www.web2py.com/init/default/download.
>>>>>>>
>>>>>>> Interestingly, gluon/packages/dal/.git is a file rather than a 
>>>>>>> directory.  It contains this:
>>>>>>>
>>>>>>> gitdir: ../../../.git/modules/gluon/packages/dal
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> All the best,
>>>>>>>
>>>>>>> Giles
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tuesday, 1 November 2016 13:25:51 UTC, Richard wrote:
>>>>>>>>
>>>>>>>> Do you download from Github or web2py.com? The later you shouldn't 
>>>>>>>> have this issue... If you get web2py from git, it normal that you have 
>>>>>>>> .git/ for web2py and for dal as dal have been extract from web2py and 
>>>>>>>> is a 
>>>>>>>> project of it own so it is include in the web2py repository as a 
>>>>>>>> submodule.
>>>>>>>>
>>>>>>>> Richard
>>>>>>>>
>>>>>>>> On Tue, Nov 1, 2016 at 8:50 AM, Giles Thomas <giles@gmail.com> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Someone pointed out to us that web2py has a .git file in 
>>>>>>>>> gluon/packages/dal/.git.  This can cause errors when you try to 
>>>>>>>>> initialize 
>>

[web2py] Re: scheduler creates "zombie" processes

[Niphlod]

I'd say there are a LOT of strange things going on on your system, since 
you're reporting several different issues that nobody ever faced and all in 
the last week.

zombie processes shouldn't be there unless you killed improperly a worker 
process. 
Python can't really do anything about it, and that's the way there's a 
specific API to kill (or terminate) a worker.

On Wednesday, November 2, 2016 at 10:53:58 AM UTC+1, Erwn Ltmann wrote:
>
> Dear all,
>
> I'm astonished about a lot of processes as sub process of scheduler worker 
> are not finished.
>
> pstree -p 16731
>  
>>
>> bash(16731)---python2.7(24545)-+-python2.7(24564)---{python2.7}(24565)
>>|-python2.7(24572)-+-python2.7(1110)
>>|  |-python2.7(8647)
>>|  |-python2.7(11747)
>>|  |-python2.7(14117)
>>|  |-python2.7(14302)
>>
>
> The 16731 is my shell I started the scheduler with four worker:
>
> w2p -K arm:ticker,arm,arm,arm
>>
>
> The pid 24564 is the ticker worker (only hold the ticker) and 24572 one of 
> three standard worker which has to process my task's function.
>
> My first focus was on the function itself. But, if I clip the function 
> ('return True' at start point) the zombies were already there. My next 
> analyze step was to show the pid at the start point of 'executor' function 
> of scheduler.py. In case of zombie processes I never reach this debug 
> point. Next I printed out the list of zombie processes 
> (multiprocessing.active_children()) at the exit point of tasks which passed 
> the timeout (see function async). It's the point in the scheduler code 
> where 'task timeout' is printing out. The timeout is clear because of a 
> process which never returns a result. But, how is it possible?
>
> Here's the list of my extra debug line in function async's timeout part:
>
> 09:09:47.752 [24576] Process-4:488,
>> 09:14:28.907 [24576] Process-4:488, Process-4:1125,
>> 09:15:59.526 [24576] Process-4:488, Process-4:1125, Process-4:1301,
>> 09:20:35.924 [24576] Process-4:488, Process-4:1880, Process-4:1125, 
>> Process-4:1301,
>>
>
> Why did the 'executor' function never process the code?
>
> def async(self, task): 
>
>   ... 
>
>   out = multiprocessing.Queue()
>>   queue = multiprocessing.Queue(maxsize=1)
>>   p = multiprocessing.Process(target=executor, args=(queue, task, out))
>
>   ...
>>   if p.is_alive():
>>p.terminate()
>>logger.debug('  +- Zombie (%s)' % 
>> multiprocessing.active_children()) 
>>
>
> And here the extra line in executor:
>
> def executor(queue, task, out):
>>   """The function used to execute tasks in the background process."""
>>   logger.debug('task started PID:%s -> %s' % 
>> (os.getppid(),os.getpid()))
>
>   ... 
>>
>
> Of course, I have to stress the scheduler to become zombies. The rate is 1 
> of 1000. In my case 25 times each hour!
>
> Can any body clarify this? May it's concerning pure python.
>
> Thx,
> Erwn
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: How to start a scheduler task from a module?

[Niphlod]

if the scheduler is already defined in the db, just use Scheduler(the_uri. 
migrate=False) . there shouldn't be issues with queuing tasks .

On Wednesday, November 2, 2016 at 11:17:00 AM UTC+1, mweissen wrote:
>
> Thank you, but it did not work, because the module program does not use 
> the models.
>
> More details:
>
> The module contains a small smtp-server. This program writes incoming 
> emails to a table "db.emails".  This part works very well. Now I want to 
> start a scheduler action to analyze each email and to do something with 
> every email. The scheduler is necessary, because every action with an email 
> may consume some time to finish.
>
>
> 2016-11-02 1:05 GMT+01:00 Dave S :
>
>>
>>
>> On Tuesday, November 1, 2016 at 2:42:28 PM UTC-7, mweissen wrote:
>>>
>>> I want to start a task from a module. (The module contains a small 
>>> smtp-server.)
>>> I could it with a db.scheduler_task.insert(...), but I want to use 
>>> myschedule.queue_task(...)
>>>
>>> ​What would be the correct way to define "myschedule" ? Something like
>>>
>>> myschedule = Scheduler(current.db)​
>>>
>>>
>>> ​Regards, Martin​
>>>
>>
>>
>> My first thought is that you define myschedule in your model file(s), and 
>> then in whatever controller calls your module, add it to current and pass 
>> it that way, or make it an argument of your module's function.  I don't see 
>> any reason NOT to use queue_task, and inserting directly into the table is 
>> subject Change Without Notice.
>>
>> /dps
>>
>>
>>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: web2py for windows server. IIS or Nginx?

[Niphlod]

*nix --> nginx with uwsgi
windows --> iis with fastcgi



On Tuesday, November 1, 2016 at 8:01:22 PM UTC+1, Omi Chiba wrote:
>
> Thank you! I will stick to Apache/wsgi for now. 
>
> On Tuesday, November 1, 2016 at 1:54:37 PM UTC-5, Jim S wrote:
>>
>> I think the thread you referenced was one discussing deployment on Ubuntu 
>> where yes, nginx/uwsgi is preferred.  But, the Windows environment is 
>> different (in my opinion) since the Windows nginx is still considered 
>> 'beta'.   I wouldn't feel that comfortable using it.  (that said, I do use 
>> many other 'beta' projects)
>>
>> If Apache/mod_wsgi is really frowned upon, should it be noted in the book?
>>
>> @omi - I migrated from Windows/Apache/mod_wsgi a while back to 
>> Ubuntu/nginx/uwsgi.  I think that at that time I switched from using pyodbc 
>> to pypyodbc.  I too access MS SQL servers from my ubuntu box using ODBC 
>> (along with IBM AS/400 databases).  It works very well for me.
>>
>> -Jim
>>
>>
>> On Tuesday, November 1, 2016 at 1:33:53 PM UTC-5, Richard wrote:
>>>
>>> I would say, we don't not don't support it, we just don't maintain a 
>>> web2py setup script with Apache... I think the decision was to reduce the 
>>> number of setup script to the bare minium to only the one web2py-devs are 
>>> willing to maintain...
>>>
>>> That been said, I am sure that if you or someone else take owner ship to 
>>> update and maintain Apache setup script because it important for you we 
>>> will bring it back in the scripts folder... But I wouldn't take that path 
>>> before someone demonstrate commitment to the task as we don't want to get 
>>> back stuff that will not be maintain in years in the repo... I guess you 
>>> can set your own github repo to demonstrate your commitment and help the 
>>> community though, and it could be reference somewhere appropriate in the 
>>> book.
>>>
>>> Richard
>>>
>>> On Tue, Nov 1, 2016 at 2:22 PM, Dave S <snide...@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Tuesday, November 1, 2016 at 7:51:26 AM UTC-7, Omi Chiba wrote:
>>>>>
>>>>> Thank you. I thought the Massimo's comment below and he  also 
>>>>> mentioned somewhere we don't want to support Apache anymore... that's why 
>>>>> I 
>>>>> was nervous. I was thinking to your direction (Moving to Ubuntu) but I 
>>>>> use 
>>>>> pyodbc to connect Microsoft SQL Server and DB2, also python-ldap.. so not 
>>>>> sure if it works the same way.
>>>>>
>>>>>
>>>>> "P.S. I stand by Niphlod. He did not say anything offending and his 
>>>>> comment was insightful. We do not recommend apache+mod_wsgi because there 
>>>>> are better ways (nginx+uwsgi)."
>>>>>
>>>>>
>>>> If you have a working Apache configuration, that's an argument for 
>>>> staying with it [caveats follow].  Part of the deprecating is because 
>>>> Apache configuration is delicate, complicated, and [reportedly] the 
>>>> documentation isn't always helpful.  If you're beyond that stage, that's 
>>>> one objection overcome.  The caveats: there is some concern that Apache 
>>>> security updates may be frequent and patching may be delicate and 
>>>> complicated [it's been around long enough that may have an "organic" 
>>>> structure].
>>>>
>>>> I think Niphlod has run both IIS and nginx on Windows, and nginx on his 
>>>> linux systems, but I'd have to go back through his posts to be sure of 
>>>> that.
>>>>
>>>> /dps
>>>>
>>>>  
>>>>
>>>>> On Tuesday, November 1, 2016 at 9:39:17 AM UTC-5, Jim S wrote:
>>>>>>
>>>>>> I haven't seen anything about Apache no longer supported.  Did I miss 
>>>>>> something?
>>>>>>
>>>>>> To my knowledge, nginx is not considered 'production ready' on 
>>>>>> Windows.  See the first paragraph here:  
>>>>>> http://nginx.org/en/docs/windows.html
>>>>>>
>>>>>> I think Apache is the way to go.   
>>>>>> http://web2py.com/books/default/chapter/29/13/deployment-recipes#Apache-and-mod_wsgi
>>>>>>
>>>>>> For me though, I've moved all of my production servers to Ubuntu with 
>>>>>> ng

Re: [web2py] .git file in gluon/packages/dal/.git

[Niphlod]

it's perfectly normal as pydal is a subrepo (it's in the readme, too)

On Tuesday, November 1, 2016 at 5:43:44 PM UTC+1, Giles Thomas wrote:
>
> Thanks, Richard.  
>
> Regarding why -- I'm talking about the version that will be installed by 
> default when someone creates a web2py app on the hosting platform my 
> company provides, PythonAnywhere -- which would also affect the version 
> used by the "Try it now online" link on the front page of www.web2py.com. 
>  I agree that doing version control the web2py way is probably more 
> sensible than making the whole web2py folder a git repo, but if people do 
> want to follow the git route, it would be better if we didn't have that 
> "trap" in there for them :-)
>
>
> All the best,
>
> Giles
>
>
> On Tuesday, 1 November 2016 16:13:22 UTC, Richard wrote:
>>
>> Sorry, you right in there is gluon/packages/dal/.git, I thought you were 
>> speaking of a .git/ folder at the root of web2py which there is not...
>>
>> I guess you can delete it if it cause problem, but if you have issue with 
>> it, it means you try to init you git repo over the entire web2py folder?
>>
>> Why don't you just version control your app? which is what we usually 
>> do...
>>
>> Thanks
>>
>> Richard
>>
>> On Tue, Nov 1, 2016 at 10:51 AM, Giles Thomas  
>> wrote:
>>
>>> The one labelled "For normal users", so presumably stable.  The VERSION 
>>> file says:
>>>
>>> Version 2.14.6-stable+timestamp.2016.05.10.00.21.47
>>>
>>>
>>> All the best,
>>>
>>> Giles
>>>
>>> On Tuesday, 1 November 2016 14:48:03 UTC, Richard wrote:

 Nightly or stable? I don't have it in my last build 2.14.6 I think... 

 Richard

 On Tue, Nov 1, 2016 at 10:42 AM, Giles Thomas  
 wrote:

> This is in a download from web2py.com -- specifically, the "Source 
> code" download from http://www.web2py.com/init/default/download.
>
> Interestingly, gluon/packages/dal/.git is a file rather than a 
> directory.  It contains this:
>
> gitdir: ../../../.git/modules/gluon/packages/dal
>
>
>
> All the best,
>
> Giles
>
>
>
>
> On Tuesday, 1 November 2016 13:25:51 UTC, Richard wrote:
>>
>> Do you download from Github or web2py.com? The later you shouldn't 
>> have this issue... If you get web2py from git, it normal that you have 
>> .git/ for web2py and for dal as dal have been extract from web2py and is 
>> a 
>> project of it own so it is include in the web2py repository as a 
>> submodule.
>>
>> Richard
>>
>> On Tue, Nov 1, 2016 at 8:50 AM, Giles Thomas  
>> wrote:
>>
>>> Hi all,
>>>
>>> Someone pointed out to us that web2py has a .git file in 
>>> gluon/packages/dal/.git.  This can cause errors when you try to 
>>> initialize 
>>> your project as a git repo.  Is there a specific reason for it to be 
>>> there?  Or is it an artefact of the packaging procedure?  We're 
>>> considering 
>>> removing it from the source that we install for users on 
>>> PythonAnywhere, 
>>> but wanted to check first to make sure that we're not going to break 
>>> anything or cause problems for our users.
>>>
>>>
>>> All the best,
>>>
>>> Giles
>>> -- 
>>> Giles Thomas 
>>>
>>> PythonAnywhere: Develop and host Python from your browser
>>> 
>>>
>>> A product from PythonAnywhere LLP
>>> 17a Clerkenwell Road, London EC1M 5RD, UK
>>> VAT No.: GB 893 5643 79
>>> Registered in England and Wales as company number OC378414.
>>> Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK 
>>>
>>>
>>> -- 
>>> Resources:
>>> - http://web2py.com
>>> - http://web2py.com/book (Documentation)
>>> - http://github.com/web2py/web2py (Source code)
>>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "web2py-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, 
>>> send an email to web2py+un...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> -- 
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> --- 
> You received this message because you are subscribed to the Google 
> Groups "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to web2py+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

 -- 
>>> Resources:
>>> - http://web2py.com
>>> - 

[web2py] Re: how to keep mysql connection alive during scheduler task running

[Niphlod]

there's nothing web2py can do if the database drops the connection without 
signalling it. 
If your diagnosis is correct, however, you can circumvent the issue using 
sync_output < 4 min, just to keep the connection alive.

On Sunday, October 30, 2016 at 3:32:02 AM UTC+1, Saint Havy wrote:
>
> Hi,
>  I used  web2py to run scheduled task which execute shell commands, 
> the code of my task as below:
> p = subprocess.Popen(command_shell,
>  shell=True,
>  stdout=subprocess.PIPE,
>  stderr=subprocess.PIPE)
> stdout, stderr = p.communicate()
> return p.returncode
> If my database is sqlite, everything is ok, but if my database is 
> mysql, then the task will not be completed when the execution time of shell 
> command is longer than 4 minutes. After my task is finished, the status of 
> scheduler_run will still be 'RUNNING', and the status of scheduler_task 
> will be 'QUEUED'. 
> I found that scheduler use a thread to create scheduler_run, and 
> update status of scheduler_run when the task is finished. The database 
> connection of the thread will be broken if the time exceeded 4 minutes and 
> no any exception will the thrown, so the status of scheduler_run can not be 
> updated. But,variable 'wait_timeout' and 'interact_timeout' of my mysql 
> server is 28800, I think the connection should keep alive in 8 hours, So, 
> is this a issue? Have it a workaround solution ?
> My database is mysql 5.5.50-0ubuntu0.14.04.1, the version of web2py is 
> 2.14.6-stable+timestamp.2016.05.10.00.21.47, they are installed in 2 VMs 
> based Azure. And, I tried MySQLdb, can't work too.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: DEADLOCKs between two or more scheduler worker

[Niphlod]

sorry, but it doesn't really make sense. 
You're executing twice the same command (the call enclosed in len() and the 
actual .delete() call), which is the counter-arg for relaxing a pressured 
database environment. 

On Monday, October 31, 2016 at 2:04:24 PM UTC+1, Erwn Ltmann wrote:
>
> Hi,
>
> thank you for your reply.
>
> @Pierre: MariaDB (in my case) handled deadlocks automaticly too. Good to 
> known, I don't have to be worry about that.
>
> @Niphlod: I tried to beef up my database host. No effects. Another 
> suggestion is to prevent the cases for such situation. I did it by an 
> another extra code line in your worker function send_heartbeat:
>
> *if len(db.executesql(dead_workers_name)) > 0:*
>>   db(
>>(st.assigned_worker_name.belongs(dead_workers_name)) &
>>(st.status == RUNNING)
>>   ).update(assigned_worker_name='', status=QUEUED)
>>   dead_workers.delete()
>>
>
>
>
> Erwn
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: question about scheduler

[Niphlod]

4. whatever you want, just avoid having large stdout (it's a general issue 
in python that can't be easily fixed). one thing is avoiding db pressure on 
the database... that means avoiding tenths of workers with thousands of 
tasks . Another is queuing a single task per hour.

5. that's perfectly normal. when you don't commit, you don't "persist" 
changes done in that transaction, but the "trace" of the changes remains: 
the database happily skips saving your data but doesn't do additional 
checks to figure out what empty spaces are around. 

transaction a --> creates a record in the table, gets a new id (let's say 
84)
transaction b --> creates a record in the table, gets a new id, that is 
what had transaction a plus one (i.e. 85)
transaction b --> creates a record in another table saving the reference to 
the previously inserted one, meaning "85" gets stored 
transaction a --> rollback()s
transaction b --> saves

for your mind-mapped method to work, the database discarding transaction a 
should also remember to rewrite what transaction b has already written and 
properly persisted. As you can imagine, it's a no go pretty much for 
everybody.

On Friday, October 28, 2016 at 12:30:34 AM UTC+2, 黄祥 wrote:
>
> thanks all
> 1. got it
> 2. already tested it, the problem is, if put all tasks in 1 controller, 
> when 1 task is failed, then run queue task, another task that have status 
> is queued will get doubled.
> 3. going to wait for the cronlike feature in stable version
>
> *another question:*
> 4. is it best practice to have shell script execute in web2py scheduler? 
> it is run well but i've seen in this forum many developers talk about avoid 
> to hit the database as much as possible, because shell script can run by os 
> (outside of web2py app)
> *e.g.*
> *static/scripts/backup.sh*
> rsync -zavur ~/web2py/applications/test ~/backup/webapp/
>
> *models/scheduler.sh*
> def backup_app():
> command = "applications/test/static/scripts/backup.sh"
> proc = subprocess.Popen(command, stdout = subprocess.PIPE, stderr = 
> subprocess.PIPE)
> output, err = proc.communicate()
> test_event.onexecute_event() # insert into db.auth_event
> db.commit()
> 5. another strange things about db.commit() in scheduler for above code, 
> when i comment db.commit(), the table auth_event is got nothing, but the id 
> is jump
> *e.g.*
> latest id on table auth_event : 10
> execute queue task for backup_app without db.commit, check latest id on 
> table auth_event : 10
> execute queue task for backup_app with db.commit, check latest id on table 
> auth_event : 12
>
> as you can see the id of auth_event is jump from 10 to 12 (don't have id 
> 11), it seems (i assume) that when execute without db.commit, it got there 
> but with no data, not sure what's happen for that case, any idea?
>
> thanks and best regards,
> stifan
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: DEADLOCKs between two or more scheduler worker

[Niphlod]

the only thing you can do is either beefing up the database instance (less 
deadlocks because of faster execution of queries) or lower the db pressure 
(lower number of workers, higher heartbeat).

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: tasks hierarchy

[Niphlod]

hum...how can cat3 be queued if cat3 depends on cat2 being executed ? 

Or what you wanted to know is how to give a higher priority to cat1 tasks 
instead of cat2 tasks (i.e. they are not related, it's just that you prefer 
cat2 remaining in the queue a bit longer if there are cat1 tasks)?
In that case, you should organize tasks into groups (i.e. cat1 with 
"high_prio", cat2 with "normal_prio", cat3 with "low_prio"), and then start 
the workers with group_names.

Although the scenario for such things requires a bit of workers (i.e. 
10-15) and to start 7 workers with high_prio, 2 with "normal_prio" and 1 
with "low_prio"

i.e. -K 
appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:normal_prio,appname:normal_prio,appname:low_prio

and that you can make the scheduler know that it should assign more tasks 
of high_prio than normal_prio, but with a lower number of workers e.g.

-K 
appname:high_prio:high_prio:high_prio:normal_prio:normal_prio:low_prio,appname:high_prio

you'd get what I call a "proportional dispatch"meaning it would 
obviously "slim" the task queue of "high_prio" before "low_prio" just 
giving more chances to high_prio tasks to be executed vs low_prio. All this 
works best if you are in the situation of having a long queue of tasks 
ready to be processed ( start_time <= request.now).

If you instead want to prioritize "scientificly" a little known fact (but 
it's shown in the scheduler test app and there's a regression test for it) 
is that tasks are picked up ordered by next_start_time. So, again, if you 
are queuing tasks that should be processed ASAP, just queue high_prio (your 
"cat1") with next_run_time = request.now - datetime.timedelta(days=90), 
normal_prio with next_run_time=request.now - datetime.timedelta(days=60) 
and low_prio with request.now.  

On Wednesday, October 26, 2016 at 4:01:07 PM UTC+2, Pierre wrote:
>
> yes i am aware of the scheduler new deps feature
> this would require to know every single task id to build the parent 
> children relation
> I don't know what's in the queue, tasks are queued in arbitrary order and 
> the queue evolves constantly so i can't scan single task ids
>
> alltogether the scheduler does a pretty good job
>
> I encounter this only once :
>
> *2016-10-25 14:55:28,137 - web2py.scheduler.hp533#4326 - ERROR - Error 
> coordinating TICKER  *
>
> is this some kind of fatal error ?
>
>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: tasks hierarchy

[Niphlod]

BTW: "error coordinating ticker" happening seldomly is not a big issue: for 
that round no workers were eligible to be tickers (i.e. dispatchers) but 
the next round will elect one.



-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: question about scheduler

[Niphlod]

1. they get deleted because in the test app the controller showing all the 
details just grabs every record in all tables. You don't need (nor want) to 
clear everything every time you queue something...else it won't be a queue 
:P
2. the deal is should the first task run alongside the second or their 
execution needs to be serial (i.e. one at a time) ? if they can go 
parallel, queue two separate tasks. if they should be serialized, go for 
two tasks with a JOB coordinating the execution only if you need to spread 
several "two tasks groups". Functionally it's exactly the same. With lots 
of "two tasks" istructions, you may get better parallelization having them 
separated rather than a big task executing two functions
3. it's not in the latest stable but is already in trunk (and I use it in 
production for quite a while) the cronline feature. Basically instead of 
queuing with *start_time *and *period *you queue with *start_time *and a 
*cronline* parameter that is in cron format. 

e.g. cronline='2 3 28 * *' will execute the task at 03:02 every 28th of any 
given month.


-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.