[web2py] revert to previous version of a record while using enable_record_versioning
Hi, I am using "auth.enable_record_versioning" to track the version of each record's update for my database tables. It works great now to keep different previous versions of a record in the "*_archive" tables. For my application, I also need features to allow some users to revert one record back to one of the previous versions. I am wondering whether it is already implemented as a function for versioning-enabled records. Such as: db[table1][id1].revert[id_archive] Here "db[table1][id1]" is one record in "table1" with id as "id1". And "id_archive" is the id of the previous record in the "*_archive" table I want to revert back to. Thank you! -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Script timed out before returning headers for apache server when scheduler task is running
Hi, I am using web2py as framework for my web application and using the built-in scheduler for the user tasks. They are working fine for more than one years. During the recent several days, I found that whenever I start a task to scheduler, the apache web server is not working, it is stuck in somewhere and give 500 internal server error. If I kill the task process or wait until the task completed, then the apache server goes back to normal. I checked the error log file of apache server, and the error message is "Script timed out before returning headers: wsgihandler.py". I think the scheduler task run is independent of the apache web server and just run in background in parallel. I don't know what could be the reason for this problem. The only thing I changed for the scheduler is the change of migrate from True to False when I initiate the scheduler: scheduler = Scheduler(db,dict(run=run),migrate=False) Did anyone encounter similar problem before? I will be very appreciated if someone can provide me a hint to solve this problem. Thanks very much! -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] change soap wsdl targetnamespace
Hi,
I want to change the target namespace to http://tempuri.org/; in the SOAP
WSDL xml file.
From the pysimplesoap's server.py, I see that I can specify it in
SoapDispatcher
class with namespace parameter. But I wonder how can I specify it in web2py
controller's @service.soap decorator.
From the code of /gluon/tools.py, it seems that the namespace could be
given by setting response.namespace
def serve_soap(self, version=1.1):
try:
from gluon.contrib.pysimplesoap.server import SoapDispatcher
except:
return pysimplesoap not installed in contrib
request = current.request
response = current.response
procedures = self.soap_procedures
location = %s://%s%s % (
request.env.wsgi_url_scheme,
request.env.http_host,
URL(r=request, f=call/soap, vars={}))
*namespace = 'namespace' in response and response.namespace or
location*
documentation = response.description or ''
dispatcher = SoapDispatcher(
name=response.title,
location=location,
action=location, # SOAPAction
namespace=namespace,
prefix='pys',
documentation=documentation,
ns=True)
I wonder how I can set response.namespace directly from web2py controller.
A working example will be very nice.
Thanks!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: change soap wsdl targetnamespace
Yeah, you are right. Stupid question. I am not familiar with how response
works.
def call():
response.namespace = 'http://tempuri.org/'
return service()
This is enough in call() function.
Thanks!
On Thursday, June 11, 2015 at 3:18:13 PM UTC-4, Niphlod wrote:
isn't a simple
def what_you_serve():
..foo bar
response.namespace = 'http://tempuri.org/'
return dict(a=1)
working ?
On Thursday, June 11, 2015 at 6:36:31 PM UTC+2, Pengfei Yu wrote:
Hi,
I want to change the target namespace to http://tempuri.org/; in the
SOAP WSDL xml file.
From the pysimplesoap's server.py, I see that I can specify it in
SoapDispatcher
class with namespace parameter. But I wonder how can I specify it in
web2py controller's @service.soap decorator.
From the code of /gluon/tools.py, it seems that the namespace could be
given by setting response.namespace
def serve_soap(self, version=1.1):
try:
from gluon.contrib.pysimplesoap.server import SoapDispatcher
except:
return pysimplesoap not installed in contrib
request = current.request
response = current.response
procedures = self.soap_procedures
location = %s://%s%s % (
request.env.wsgi_url_scheme,
request.env.http_host,
URL(r=request, f=call/soap, vars={}))
*namespace = 'namespace' in response and response.namespace or
location*
documentation = response.description or ''
dispatcher = SoapDispatcher(
name=response.title,
location=location,
action=location, # SOAPAction
namespace=namespace,
prefix='pys',
documentation=documentation,
ns=True)
I wonder how I can set response.namespace directly from web2py
controller. A working example will be very nice.
Thanks!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: change soap wsdl targetnamespace
yeah, but I have several different functions in the SOAP service. if I set
it in call() function, i just need to set it once. Otherwise I need to set
this for each called function.
On Thursday, June 11, 2015 at 3:43:17 PM UTC-4, Niphlod wrote:
even in the called function should be enough.
On Thursday, June 11, 2015 at 9:39:32 PM UTC+2, Pengfei Yu wrote:
Yeah, you are right. Stupid question. I am not familiar with how response
works.
def call():
response.namespace = 'http://tempuri.org/'
return service()
This is enough in call() function.
Thanks!
On Thursday, June 11, 2015 at 3:18:13 PM UTC-4, Niphlod wrote:
isn't a simple
def what_you_serve():
..foo bar
response.namespace = 'http://tempuri.org/'
return dict(a=1)
working ?
On Thursday, June 11, 2015 at 6:36:31 PM UTC+2, Pengfei Yu wrote:
Hi,
I want to change the target namespace to http://tempuri.org/; in the
SOAP WSDL xml file.
From the pysimplesoap's server.py, I see that I can specify it in
SoapDispatcher
class with namespace parameter. But I wonder how can I specify it in
web2py controller's @service.soap decorator.
From the code of /gluon/tools.py, it seems that the namespace could
be given by setting response.namespace
def serve_soap(self, version=1.1):
try:
from gluon.contrib.pysimplesoap.server import
SoapDispatcher
except:
return pysimplesoap not installed in contrib
request = current.request
response = current.response
procedures = self.soap_procedures
location = %s://%s%s % (
request.env.wsgi_url_scheme,
request.env.http_host,
URL(r=request, f=call/soap, vars={}))
*namespace = 'namespace' in response and response.namespace or
location*
documentation = response.description or ''
dispatcher = SoapDispatcher(
name=response.title,
location=location,
action=location, # SOAPAction
namespace=namespace,
prefix='pys',
documentation=documentation,
ns=True)
I wonder how I can set response.namespace directly from web2py
controller. A working example will be very nice.
Thanks!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: soap client with windows NTLM authorization
'USERPWD' and populate
it. Take a look at lines 75+.
See all those 'setopt' calls? Add in your own for HTTPAUTH and USERPWD ...
On Tuesday, June 9, 2015 at 7:27:55 AM UTC-7, Pengfei Yu wrote:
Thanks for your reply! I used the proxy because I checked from the source
code that proxy is the only place I can pass my username and password to
pycurl from SoapClient class.
I can connect to this SOAP service directly with pycurl using the code
you provided. But I want to create a SoapClient object based on this SOAP
service. I still cannot figure it out yet.
On Monday, June 8, 2015 at 5:28:08 PM UTC-4, Derek wrote:
a simple monkey patch will do you. I would suggest you don't import into
the base namespace though.
import gluon.contrib.pysimplesoap.client as ssClient
then do the monkey...
ssClient.Http = set_http_wrapper(library='pycurl')
and use it like normal.
I don't get why you are trying to use a proxy?
import pycurl
name='bob'
pwd='pwd1'
url=https://mywebservice;
curl = pycurl.Curl()
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
curl.setopt(pycurl.HTTPAUTH, pycurl.HTTPAUTH_NTLM)
curl.setopt(pycurl.USERPWD, {}:{}.format(name, pwd))
curl.perform()
curl.close()
On Monday, June 8, 2015 at 11:45:44 AM UTC-7, Pengfei Yu wrote:
Hi Derek,
Thanks for your reply! I saw similar source code as well. But there is
no document how to set it up using pysimplesoap. Could you provide an
example?
I tried to use following, but it cannot work.
import sys,time
sys.path.append(/home/www-data/web2py)
import pprint
from gluon.contrib.pysimplesoap.client import *
from gluon.contrib.pysimplesoap.transport import *
user = 'XXX'
password = ***
proxy={'proxy_user':user,'proxy_pass':password}
Http = set_http_wrapper(library='pycurl')
client=SoapClient(wsdl=https://54.153.5.133:53441/ForAGISService?wsdl;
,proxy=proxy)
Thanks!
On Monday, June 8, 2015 at 12:18:38 PM UTC-4, Derek wrote:
looks like pycurl is supported by pysimplesoap. That supports NTLM.
See line 67.
https://code.google.com/p/pysimplesoap/source/browse/pysimplesoap/client.py?r=6ed06397b4f0c1894156ee5d0a1c165f80ed6a68
On Monday, June 8, 2015 at 7:28:39 AM UTC-7, Pengfei Yu wrote:
Hi,
I am trying to access a web service which requires windows NTLM
authorization. I am able to successfully implement it using suds python
library with following code:
from suds.transport.http import *
from suds.transport.https import WindowsHttpAuthenticated
from suds.client import *
import time
sampleID = AA
user = 'XXX'
password = ***
url = https://54.153.5.133:53441/ForAGISService?wsdl;
transport = WindowsHttpAuthenticated(username=user, password=password
)
client = Client(url, transport=transport)
print List of methods for this web service:
print [method for method in client.wsdl.services[0].ports[0].methods]
print \nsample info:
print client.service.GetSampleInfoById(sampleID)
The NTLM transport is supported by python-ntlm package as mentioned
in https://fedorahosted.org/suds/wiki/Documentation#WindowsNTLM.
But I prefer to use pysimplesoap as SOAP client in my web2py
application. I wonder if there is also an feasible approach to implement
it
with pysimplesoap + python-ntlm? If someone could provide a code
example,
that will be perfect.
Thanks!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: soap client with windows NTLM authorization
Thanks for your reply! I used the proxy because I checked from the source
code that proxy is the only place I can pass my username and password to
pycurl from SoapClient class.
I can connect to this SOAP service directly with pycurl using the code you
provided. But I want to create a SoapClient object based on this SOAP
service. I still cannot figure it out yet.
On Monday, June 8, 2015 at 5:28:08 PM UTC-4, Derek wrote:
a simple monkey patch will do you. I would suggest you don't import into
the base namespace though.
import gluon.contrib.pysimplesoap.client as ssClient
then do the monkey...
ssClient.Http = set_http_wrapper(library='pycurl')
and use it like normal.
I don't get why you are trying to use a proxy?
import pycurl
name='bob'
pwd='pwd1'
url=https://mywebservice;
curl = pycurl.Curl()
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
curl.setopt(pycurl.HTTPAUTH, pycurl.HTTPAUTH_NTLM)
curl.setopt(pycurl.USERPWD, {}:{}.format(name, pwd))
curl.perform()
curl.close()
On Monday, June 8, 2015 at 11:45:44 AM UTC-7, Pengfei Yu wrote:
Hi Derek,
Thanks for your reply! I saw similar source code as well. But there is no
document how to set it up using pysimplesoap. Could you provide an example?
I tried to use following, but it cannot work.
import sys,time
sys.path.append(/home/www-data/web2py)
import pprint
from gluon.contrib.pysimplesoap.client import *
from gluon.contrib.pysimplesoap.transport import *
user = 'XXX'
password = ***
proxy={'proxy_user':user,'proxy_pass':password}
Http = set_http_wrapper(library='pycurl')
client=SoapClient(wsdl=https://54.153.5.133:53441/ForAGISService?wsdl;,
proxy=proxy)
Thanks!
On Monday, June 8, 2015 at 12:18:38 PM UTC-4, Derek wrote:
looks like pycurl is supported by pysimplesoap. That supports NTLM. See
line 67.
https://code.google.com/p/pysimplesoap/source/browse/pysimplesoap/client.py?r=6ed06397b4f0c1894156ee5d0a1c165f80ed6a68
On Monday, June 8, 2015 at 7:28:39 AM UTC-7, Pengfei Yu wrote:
Hi,
I am trying to access a web service which requires windows NTLM
authorization. I am able to successfully implement it using suds python
library with following code:
from suds.transport.http import *
from suds.transport.https import WindowsHttpAuthenticated
from suds.client import *
import time
sampleID = AA
user = 'XXX'
password = ***
url = https://54.153.5.133:53441/ForAGISService?wsdl;
transport = WindowsHttpAuthenticated(username=user, password=password)
client = Client(url, transport=transport)
print List of methods for this web service:
print [method for method in client.wsdl.services[0].ports[0].methods]
print \nsample info:
print client.service.GetSampleInfoById(sampleID)
The NTLM transport is supported by python-ntlm package as mentioned in
https://fedorahosted.org/suds/wiki/Documentation#WindowsNTLM.
But I prefer to use pysimplesoap as SOAP client in my web2py
application. I wonder if there is also an feasible approach to implement
it
with pysimplesoap + python-ntlm? If someone could provide a code example,
that will be perfect.
Thanks!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: soap client with windows NTLM authorization
Hi Derek,
Thanks for your reply! I saw similar source code as well. But there is no
document how to set it up using pysimplesoap. Could you provide an example?
I tried to use following, but it cannot work.
import sys,time
sys.path.append(/home/www-data/web2py)
import pprint
from gluon.contrib.pysimplesoap.client import *
from gluon.contrib.pysimplesoap.transport import *
user = 'XXX'
password = ***
proxy={'proxy_user':user,'proxy_pass':password}
Http = set_http_wrapper(library='pycurl')
client=SoapClient(wsdl=https://54.153.5.133:53441/ForAGISService?wsdl;,
proxy=proxy)
Thanks!
On Monday, June 8, 2015 at 12:18:38 PM UTC-4, Derek wrote:
looks like pycurl is supported by pysimplesoap. That supports NTLM. See
line 67.
https://code.google.com/p/pysimplesoap/source/browse/pysimplesoap/client.py?r=6ed06397b4f0c1894156ee5d0a1c165f80ed6a68
On Monday, June 8, 2015 at 7:28:39 AM UTC-7, Pengfei Yu wrote:
Hi,
I am trying to access a web service which requires windows NTLM
authorization. I am able to successfully implement it using suds python
library with following code:
from suds.transport.http import *
from suds.transport.https import WindowsHttpAuthenticated
from suds.client import *
import time
sampleID = AA
user = 'XXX'
password = ***
url = https://54.153.5.133:53441/ForAGISService?wsdl;
transport = WindowsHttpAuthenticated(username=user, password=password)
client = Client(url, transport=transport)
print List of methods for this web service:
print [method for method in client.wsdl.services[0].ports[0].methods]
print \nsample info:
print client.service.GetSampleInfoById(sampleID)
The NTLM transport is supported by python-ntlm package as mentioned in
https://fedorahosted.org/suds/wiki/Documentation#WindowsNTLM.
But I prefer to use pysimplesoap as SOAP client in my web2py application.
I wonder if there is also an feasible approach to implement it with
pysimplesoap + python-ntlm? If someone could provide a code example, that
will be perfect.
Thanks!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] soap client with windows NTLM authorization
Hi, I am trying to access a web service which requires windows NTLM authorization. I am able to successfully implement it using suds python library with following code: from suds.transport.http import * from suds.transport.https import WindowsHttpAuthenticated from suds.client import * import time sampleID = AA user = 'XXX' password = *** url = https://54.153.5.133:53441/ForAGISService?wsdl; transport = WindowsHttpAuthenticated(username=user, password=password) client = Client(url, transport=transport) print List of methods for this web service: print [method for method in client.wsdl.services[0].ports[0].methods] print \nsample info: print client.service.GetSampleInfoById(sampleID) The NTLM transport is supported by python-ntlm package as mentioned in https://fedorahosted.org/suds/wiki/Documentation#WindowsNTLM. But I prefer to use pysimplesoap as SOAP client in my web2py application. I wonder if there is also an feasible approach to implement it with pysimplesoap + python-ntlm? If someone could provide a code example, that will be perfect. Thanks! -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler workers get deleted constantly and one task is assigned to many different runs
This need to be verified with AWS's tech support I guess. The instances in
AWS region cn-north-1 are weird in many aspects from my recent
experiences...
On Sunday, May 3, 2015 at 8:18:59 AM UTC-4, Niphlod wrote:
seems that the same thing that happened a long time ago on travis-ci: glad
you worked it out, but I'm really unsure why different regions should carry
different versions of the same operating system.
On Thursday, April 30, 2015 at 6:57:08 PM UTC+2, Pengfei Yu wrote:
Hi Niphlod,
I think I solve the problem. Followed by my last post, the issure is in
shared memory for the multiprocessing module of python. According to the
solution on this link (
http://stackoverflow.com/questions/2009278/python-multiprocessing-permission-denied
), I let none being mounted on /dev/shm/ and the problem is fixed.
The interesting thing is that in EC2 instance of US regions, I don't need
to do this mounting, and it works fine for the multiprocessing, but for the
instance in China region, I need to do that.
Thanks for all your helps!
Pengfei
On Thursday, April 30, 2015 at 8:47:48 AM UTC-4, Niphlod wrote:
On Wednesday, April 29, 2015 at 11:53:56 PM UTC+2, Pengfei Yu wrote:
Hi Niphlod and Dave,
I think the worker is deleted because the the last heartbeat doesn't
change for a while. When I comment out the line #dead_workers.delete()
in
gluon/schdeuler.py, the task is stuck as QUEUED without new run record
created. And the worker supposed to be deleted with the last_heartbeat
doesn't change. When I keep this line dead_workers.delete(), the
situation remain the same as my original email.
a worker that can't piggyback its presence NEEDS to be assumed as dead.
I think the task is never processed, since I added a single test line
as os.system('touch some_folder/test.log') at the beginning of the
task
function, but this test.log file is never created in the server. I guess
the worker is died right after it is assigned a task, even before the task
is being processed, and then the task is QUEUED again.
in this case, you won't have any scheduler_run records. Those are
created when the task gets picked up.
I have the configuration in /etc/init/web2py-scheduler,conf, and the
service is running, that is why new worker are pop out when previous one
dies.
Perfect, but maybe you should monitor if they ALWAYS fail and avoid
restarting them ?!?!!?
When I set timeout as 10s for the task, it is still the same as before,
multiple runs are generated for the same task and they are not ended
though
the assigned workers are died. I guess for the task, the time from RUNNING
to QUEUED again is very short.
This seems to point out that the problem isn't a really long task that
gets stuck, but rather that your task even before the 10 seconds has
issues, or the scheduler has trying to pick that up.
One difference between the server in original region (works fine) and
the current region is that in the current region, I use the port 8000 for
the HTTPS access, instead of the default 443. This is because the current
one is in China and there are restrictions on HTTP/HTTPS access with port
80/8080/443. I am not sure if this will affect the workers since the
worker
name only have ip information but not port.
Shouldn't be a problem.
Niphlod,
May I ask how can I enable the DEBUG mode and check the logging? I do
see logging in the script of gluon/scheduler.py, but I don't know where I
can find the logging information in a log file.
Thanks
use logging.conf (enable DEBUG log for root and web2py, then in
consolehandler) and start the scheduler with
web2py.py -K appname -D 0
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler workers get deleted constantly and one task is assigned to many different runs
Hi Niphlod,
I think I solve the problem. Followed by my last post, the issure is in
shared memory for the multiprocessing module of python. According to the
solution on this link
(http://stackoverflow.com/questions/2009278/python-multiprocessing-permission-denied
), I let none being mounted on /dev/shm/ and the problem is fixed.
The interesting thing is that in EC2 instance of US regions, I don't need
to do this mounting, and it works fine for the multiprocessing, but for the
instance in China region, I need to do that.
Thanks for all your helps!
Pengfei
On Thursday, April 30, 2015 at 8:47:48 AM UTC-4, Niphlod wrote:
On Wednesday, April 29, 2015 at 11:53:56 PM UTC+2, Pengfei Yu wrote:
Hi Niphlod and Dave,
I think the worker is deleted because the the last heartbeat doesn't
change for a while. When I comment out the line #dead_workers.delete() in
gluon/schdeuler.py, the task is stuck as QUEUED without new run record
created. And the worker supposed to be deleted with the last_heartbeat
doesn't change. When I keep this line dead_workers.delete(), the
situation remain the same as my original email.
a worker that can't piggyback its presence NEEDS to be assumed as dead.
I think the task is never processed, since I added a single test line as
os.system('touch some_folder/test.log') at the beginning of the task
function, but this test.log file is never created in the server. I guess
the worker is died right after it is assigned a task, even before the task
is being processed, and then the task is QUEUED again.
in this case, you won't have any scheduler_run records. Those are created
when the task gets picked up.
I have the configuration in /etc/init/web2py-scheduler,conf, and the
service is running, that is why new worker are pop out when previous one
dies.
Perfect, but maybe you should monitor if they ALWAYS fail and avoid
restarting them ?!?!!?
When I set timeout as 10s for the task, it is still the same as before,
multiple runs are generated for the same task and they are not ended though
the assigned workers are died. I guess for the task, the time from RUNNING
to QUEUED again is very short.
This seems to point out that the problem isn't a really long task that
gets stuck, but rather that your task even before the 10 seconds has
issues, or the scheduler has trying to pick that up.
One difference between the server in original region (works fine) and the
current region is that in the current region, I use the port 8000 for the
HTTPS access, instead of the default 443. This is because the current one
is in China and there are restrictions on HTTP/HTTPS access with port
80/8080/443. I am not sure if this will affect the workers since the worker
name only have ip information but not port.
Shouldn't be a problem.
Niphlod,
May I ask how can I enable the DEBUG mode and check the logging? I do see
logging in the script of gluon/scheduler.py, but I don't know where I can
find the logging information in a log file.
Thanks
use logging.conf (enable DEBUG log for root and web2py, then in
consolehandler) and start the scheduler with
web2py.py -K appname -D 0
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: scheduler workers get deleted constantly and one task is assigned to many different runs
:task completed (COMPLETED)
DEBUG:web2py.scheduler.ip-20-20-10-190#15277:looping...
Thanks very much!
On Thursday, April 30, 2015 at 8:47:48 AM UTC-4, Niphlod wrote:
On Wednesday, April 29, 2015 at 11:53:56 PM UTC+2, Pengfei Yu wrote:
Hi Niphlod and Dave,
I think the worker is deleted because the the last heartbeat doesn't
change for a while. When I comment out the line #dead_workers.delete() in
gluon/schdeuler.py, the task is stuck as QUEUED without new run record
created. And the worker supposed to be deleted with the last_heartbeat
doesn't change. When I keep this line dead_workers.delete(), the
situation remain the same as my original email.
a worker that can't piggyback its presence NEEDS to be assumed as dead.
I think the task is never processed, since I added a single test line as
os.system('touch some_folder/test.log') at the beginning of the task
function, but this test.log file is never created in the server. I guess
the worker is died right after it is assigned a task, even before the task
is being processed, and then the task is QUEUED again.
in this case, you won't have any scheduler_run records. Those are created
when the task gets picked up.
I have the configuration in /etc/init/web2py-scheduler,conf, and the
service is running, that is why new worker are pop out when previous one
dies.
Perfect, but maybe you should monitor if they ALWAYS fail and avoid
restarting them ?!?!!?
When I set timeout as 10s for the task, it is still the same as before,
multiple runs are generated for the same task and they are not ended though
the assigned workers are died. I guess for the task, the time from RUNNING
to QUEUED again is very short.
This seems to point out that the problem isn't a really long task that
gets stuck, but rather that your task even before the 10 seconds has
issues, or the scheduler has trying to pick that up.
One difference between the server in original region (works fine) and the
current region is that in the current region, I use the port 8000 for the
HTTPS access, instead of the default 443. This is because the current one
is in China and there are restrictions on HTTP/HTTPS access with port
80/8080/443. I am not sure if this will affect the workers since the worker
name only have ip information but not port.
Shouldn't be a problem.
Niphlod,
May I ask how can I enable the DEBUG mode and check the logging? I do see
logging in the script of gluon/scheduler.py, but I don't know where I can
find the logging information in a log file.
Thanks
use logging.conf (enable DEBUG log for root and web2py, then in
consolehandler) and start the scheduler with
web2py.py -K appname -D 0
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] scheduler workers get deleted constantly and one task is assigned to many different runs
Hi,
I have a scheduler problem after transferring our web2py server in another
region. We used EC2 for the server and RDS for the mySql database. It works
fine in original region. Right now we move the server and the database to
another region with new AWS EC2 and new RDS. I used the web2py-scheduler
configuration to automatically initiate workers when the server is running.
In the new region, originally, we have 5 workers available. when we created
a new scheduler task waiting to be assigned to the workers, the task is
first been assigned to a worker and a run is created, then quickly this
worker is deleted and the task is queued again, and then the task is
assigned to another active worker and another run is created, then the new
assigned worker is deleted again. It happens all the time if we don't
remove the task, all the previously assigned runs are still running, they
are all for the same task (see attached db_scheduler_run.csv). But the
assigned workers are already deleted and gone from the database
(db_scheduler_worker.csv). Since we start the web2py-scheduler, every time
old workers are deleted, new workers will be generated, so in the table of
db_scheduler_worker.csv are the new workers.
I don't understand why the worker is deleted immediately once it has been
assigned to a run for a task.
In the code of gluon/scheduler.py, I found the conditions for deletion of a
worker, but I still cannot find the reason here why my workers get deleted.
self.do_assign_tasks = False if counter % 5 == 0 or mybackedstatus == PICK:
try: # delete dead workers expiration = now - datetime.timedelta( seconds=
self.heartbeat * 3) departure = now - datetime.timedelta(
seconds=self.heartbeat
* 3 * 15) logger.debug( ' freeing workers that have not sent heartbeat')
dead_workers = db( ((sw.last_heartbeat expiration) (sw.status ==
ACTIVE)) | ((sw.last_heartbeat departure) (sw.status != ACTIVE)) )
dead_workers_name = dead_workers._select(sw.worker_name) db(
(st.assigned_worker_name.belongs(dead_workers_name)) (st.status ==
RUNNING) ).update(assigned_worker_name='', status=QUEUED)
dead_workers.delete() try: self.is_a_ticker = self.being_a_ticker() except:
logger.error('Error coordinating TICKER') if self.w_stats.status == ACTIVE:
self.do_assign_tasks = True except: logger.error('Error cleaning up')
db.commit()
I think it may related to the heartbeat, but I cannot figure out why. The
server works fine in original region, but this error happens when I copy
the code using git repository from server in original region to server in
new region.
Thanks very much!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
scheduler_run.id,scheduler_run.task_id,scheduler_run.status,scheduler_run.start_time,scheduler_run.stop_time,scheduler_run.run_output,scheduler_run.run_result,scheduler_run.traceback,scheduler_run.worker_name
474,6,RUNNING,2015-04-29 16:09:55,NULL,NULL,NULL,NULL,ip-172-31-2-63#91165
475,6,RUNNING,2015-04-29 16:10:10,NULL,NULL,NULL,NULL,ip-172-31-2-63#91169
476,6,RUNNING,2015-04-29 16:10:26,NULL,NULL,NULL,NULL,ip-172-31-2-63#91173
477,6,RUNNING,2015-04-29 16:10:42,NULL,NULL,NULL,NULL,ip-172-31-2-63#91177
478,6,RUNNING,2015-04-29 16:10:58,NULL,NULL,NULL,NULL,ip-172-31-2-63#91181
479,6,RUNNING,2015-04-29 16:30:14,NULL,NULL,NULL,NULL,ip-172-31-2-63#92122
480,6,RUNNING,2015-04-29 16:30:30,NULL,NULL,NULL,NULL,ip-172-31-2-63#92126
481,6,RUNNING,2015-04-29 16:30:46,NULL,NULL,NULL,NULL,ip-172-31-2-63#92130
482,6,RUNNING,2015-04-29 16:31:01,NULL,NULL,NULL,NULL,ip-172-31-2-63#92134
483,6,RUNNING,2015-04-29 16:31:17,NULL,NULL,NULL,NULL,ip-172-31-2-63#92138
484,6,RUNNING,2015-04-29 16:31:39,NULL,NULL,NULL,NULL,ip-172-31-2-63#92173
485,6,RUNNING,2015-04-29 16:31:54,NULL,NULL,NULL,NULL,ip-172-31-2-63#92177
486,6,RUNNING,2015-04-29 16:32:10,NULL,NULL,NULL,NULL,ip-172-31-2-63#92181
487,6,RUNNING,2015-04-29 16:32:26,NULL,NULL,NULL,NULL,ip-172-31-2-63#92185
488,6,RUNNING,2015-04-29 16:32:42,NULL,NULL,NULL,NULL,ip-172-31-2-63#92189
489,6,RUNNING,2015-04-29 16:33:04,NULL,NULL,NULL,NULL,ip-172-31-2-63#92233
490,6,RUNNING,2015-04-29 16:33:19,NULL,NULL,NULL,NULL,ip-172-31-2-63#92237
491,6,RUNNING,2015-04-29 16:33:35,NULL,NULL,NULL,NULL,ip-172-31-2-63#92241
492,6,RUNNING,2015-04-29 16:33:50,NULL,NULL,NULL,NULL,ip-172-31-2-63#92245
493,6,RUNNING,2015-04-29 16:34:06,NULL,NULL,NULL,NULL,ip-172-31-2-63#92249
494,6,RUNNING,2015-04-29 16:34:28,NULL,NULL,NULL,NULL,ip-172-31-2-63#92303
495,6,RUNNING,2015-04-29 16:34:44,NULL,NULL,NULL,NULL,ip-172-31-2-63#92307
496,6,RUNNING,2015-04-29
[web2py] Re: scheduler workers get deleted constantly and one task is assigned to many different runs
Hi Niphlod and Dave,
I think the worker is deleted because the the last heartbeat doesn't change
for a while. When I comment out the line #dead_workers.delete() in
gluon/schdeuler.py, the task is stuck as QUEUED without new run record
created. And the worker supposed to be deleted with the last_heartbeat
doesn't change. When I keep this line dead_workers.delete(), the
situation remain the same as my original email.
I think the task is never processed, since I added a single test line as
os.system('touch some_folder/test.log') at the beginning of the task
function, but this test.log file is never created in the server. I guess
the worker is died right after it is assigned a task, even before the task
is being processed, and then the task is QUEUED again.
I have the configuration in /etc/init/web2py-scheduler,conf, and the
service is running, that is why new worker are pop out when previous one
dies.
When I set timeout as 10s for the task, it is still the same as before,
multiple runs are generated for the same task and they are not ended though
the assigned workers are died. I guess for the task, the time from RUNNING
to QUEUED again is very short.
scheduler_run.id
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.id
scheduler_run.task_id
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.task_id
scheduler_run.status
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.status
scheduler_run.start_time
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.start_time
scheduler_run.stop_time
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.stop_time
scheduler_run.run_output
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.run_output
scheduler_run.run_result
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.run_result
scheduler_run.traceback
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.traceback
scheduler_run.worker_name
https://54.223.188.59:8000/AGIS/appadmin/select/db?orderby=scheduler_run.worker_name
130 https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_run/130
run https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_task/4
RUNNING2015-04-29 21:41:32NoneNoneNoneNoneip-172-31-2-6...131
https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_run/131run
https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_task/4RUNNING2015-04-29
21:41:47NoneNoneNoneNoneip-172-31-2-6...132
https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_run/132run
https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_task/4RUNNING2015-04-29
21:42:03NoneNoneNoneNoneip-172-31-2-6...133
https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_run/133run
https://54.223.188.59:8000/AGIS/appadmin/update/db/scheduler_task/4RUNNING2015-04-29
21:42:18NoneNoneNoneNoneip-172-31-2-6...
I also did clean all schedule related tables in /database/ and my mySql
database, and restart from the empty scheduler related tables, but it is
still the same.
One difference between the server in original region (works fine) and the
current region is that in the current region, I use the port 8000 for the
HTTPS access, instead of the default 443. This is because the current one
is in China and there are restrictions on HTTP/HTTPS access with port
80/8080/443. I am not sure if this will affect the workers since the worker
name only have ip information but not port.
Niphlod,
May I ask how can I enable the DEBUG mode and check the logging? I do see
logging in the script of gluon/scheduler.py, but I don't know where I can
find the logging information in a log file.
Thanks!
On Wednesday, April 29, 2015 at 4:04:27 PM UTC-4, Niphlod wrote:
whoa. Thank god I put some statistics in to help with debugging..
It seems that the worker processes are handling one task and dieing in the
middle of the process.
Also, it seems that your task is never-ending.
workers that haven't a task to be processed are running fine (empty_runs
on 93164 and 93168 is 20, that means that they're quietly looping), so as
long as they don't process any task they seem to be healthy.
Every execution is done by a different worker, that seems to point towards
the previous analysis of the worker dieing in the middle of processing the
task.
Worker processes are not respawned automatically by web2py, so it's likely
you have in place some kind of supervisor that restarts them when they die.
I'd make two things: 1) setting a timeout on the task, just to see if it's
so long that it blocks somewhere (it may be leaking). 2) inspect logs of
the worker processes to see where they get stuck.
I'd also stop all the workers, start one in the console with DEBUG
logging, and see see what's going on in real-time.
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
-
[web2py] Re: ghost scheduler workers can never be deleted even the web2py-schduler is stopped.
Thanks Niphlod! It turns out that we have an data base which is shared by two duplicated instances and the other workers are from the other instance. We've fixed that. On Saturday, April 25, 2015 at 8:51:04 AM UTC-4, Niphlod wrote: there's nothing that magically insert records on the scheduler_worker table. watch the last_heartbeat column and see if it changes. If it's changing, the workers are still running somewhere and accessing your database. On Friday, April 24, 2015 at 9:24:59 PM UTC+2, Pengfei Yu wrote: Hi I am using web2py scheduler. I have a project saved in git and then I cloned this git repository to another new instance (server). In the new server, five scheduler workers (known from the ip address in the worker_name) from the old server are always there. When never I delete them, they will automatically generated soon. And it seems that they inherited the scheduler task program from the old server. After I updated the scheduler task program, it is not updated for these five ghost worker. Each time I initiate a new task, it will be assigned to these 5 workers first, and the updated task program will never be run. Could any one tell me how to git rid of these ghost workers. Thanks! -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] ghost scheduler workers can never be deleted even the web2py-schduler is stopped.
Hi I am using web2py scheduler. I have a project saved in git and then I cloned this git repository to another new instance (server). In the new server, five scheduler workers (known from the ip address in the worker_name) from the old server are always there. When never I delete them, they will automatically generated soon. And it seems that they inherited the scheduler task program from the old server. After I updated the scheduler task program, it is not updated for these five ghost worker. Each time I initiate a new task, it will be assigned to these 5 workers first, and the updated task program will never be run. Could any one tell me how to git rid of these ghost workers. Thanks! -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler: tasks are always assigned to the same running worker.
Wow! Thanks Niphlod for your kind and efficient helps. And I appreciate
your praise words to me. This is very encouraging for an amateur python-er.
In my real-word application, I used MySQL database from AWS RDS. Do you
think if it will also cause concurrent issue?
I see in the gist that only the gluon/scheduler.py file has been changed.
So is it fine if I only update this scheduler.py file from your new commit?
I tried to update this python file and it seems the worker assignment
didn't change for the toy application. Do I need to restart the web2py
application to make it work? How I can do that if it is already deployed in
EC2 instance?
Thanks!
On Thursday, December 18, 2014 5:58:35 PM UTC-5, Niphlod wrote:
ok, got it, all tests pass. PR ready at
https://github.com/web2py/web2py/pull/565
In the meantime, if you want to verify the better assignment, all you
have to do is applying this
https://github.com/web2py/web2py/pull/565/files#diff-1cbb9c488bb09b40aff4560078808c79R1091
to the scheduler currently on master, i.e. on the assign_tasks function,
add the green part.
...
#build workers as dict of groups
wkgroups = {}
for w in all_workers:
if w.worker_stats['status'] == 'RUNNING':
continue
group_names = w.group_names
..
@all: this is one of the best bug report/feature request discussion
I've seen so far:
- User asking politely legitimate questions
- an app to reproduce the behaviour with examples
- user reporting there's something wrong with actual data at hand,
without barking at the wrong tree
@PengfeiYu: it's been a real pleasure. See you
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler: tasks are always assigned to the same running worker.
Yeah, I just made it work. Since I used Apache2 to host web2py in EC2 instance, after updating the scheduler.py file, I need to restart the Apache2 server by: sudo service apache2 restart and then restart the web2py-scheduler by: sudo stop web2py-scheduler sudo start web2py-scheduler (this is assuming the /etc/init/web2py-shceduler.conf is there) Thanks! On Friday, December 19, 2014 3:20:35 PM UTC-5, Niphlod wrote: In my real-word application, I used MySQL database from AWS RDS. Do you think if it will also cause concurrent issue? nope, mysql is fine with concurrency I see in the gist that only the gluon/scheduler.py file has been changed. So is it fine if I only update this scheduler.py file from your new commit? I tried to update this python file and it seems the worker assignment didn't change for the toy application. Do I need to restart the web2py application to make it work? How I can do that if it is already deployed in EC2 instance? you can just pull in the scheduler.py file with the modifications: there's no breaking changes in the modification I did (or use this https://raw.githubusercontent.com/niphlod/web2py/scheduler/enhancement/gluon/scheduler.py). you just need to restart the scheduler to make the changes work. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler: tasks are always assigned to the same running worker.
Hi Niphlod,
I found a strange situation even using the testtask.
Here are two different scenarios. In the first scenario, the two test tasks
can be run concurrently as in my last reply. In the second scenario, the
two test tasks will be assigned to the same worker, although the worker is
already running the first task.
*Scenario 1:*
In the default controler, I use:
def index():
scheduler.queue_task(testtask,timeout=1000)
db.commit()
time.sleep(10)
scheduler.queue_task(testtask,timeout=1000)
db.commit()
return dict()
Then when I access the index page, two test tasks are created one-by-one
and are assigned to different active workers. There is overlap in their
running periods.
Scenario 2:
If I create another function in default controller to initiate one task:
def index():
return dict()
def TestTask():
scheduler.queue_task(testtask,timeout=1000)
db.commit()
redirect(URL(index))
And in the view, index.html page, I create a link to call TestTask():
h4Test_task:/h4
a href={{=URL('TestTask')}}Click to start/a
Each time I click the link, one test task will be created. But if I click
twice, the two initiated tasks are always assigned to the same worker, and
I need to wait until one task to be completed to run the other one. I am
not sure if it is a bug or not. Is there any way to make the scheduler to
assign tasks the same way as scenario 1 in the case of scenario 2?
I tried to remove and include db.commit() for both scenarios, and the
results do not change.
Thanks!
On Wednesday, December 17, 2014 8:12:37 AM UTC-5, Niphlod wrote:
ok, so it's not a problem of the algorithm of dispatching tasks of the
scheduler . Try adding small bits of your code to the test_task to pinpoint
the root cause of the original misbehaviour.
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler: tasks are always assigned to the same running worker.
Hi Niphlod, Thanks for your reply! The problem is the 5681 is taking the task 1 and it is RUNNING (See the worker stats of worker 1). But at the same time, the task is still assigned to 5681, but not two other ACTIVE workers, 5685 and 5690. I previously thought the reason might be in the status feature of worker 5681, it still shows ACTIVE although it is RUNNING the task 1 as shown in the worker stats. But it seems that the testtask case works fine. two tasks were running simultaneously by worker 5681 and 5685 (See the pictures). On Tuesday, December 16, 2014 10:49:35 AM UTC-5, Niphlod wrote: Uhm, really strange. Keep things simple to reproduce: queue tasks that are something like import time def testtask(): time.sleep(60) return 1 and see if the behaviour is reproducible. Here I don't see anything wrong: 5861 is ACTIVE (meaning there's nothing actively running on it, or it would be RUNNING), 5865 is ACTIVE too, and it's the one distributing tasks. 3 workers are recognized correctly, with 1 task still in the queue -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Java Client for web2py SOAP web service
Cool! Thanks very much for sharing, Dave!
On Tuesday, November 25, 2014 2:26:47 PM UTC-5, Dave S wrote:
On Monday, November 24, 2014 12:24:25 PM UTC-8, Pengfei Yu wrote:
Hi Dave,
Thanks very much for your reply! Could you give a more detailed example
for an implementation for Java client? A block of java codes from your Java
client talking to web2py service will be perfect. Even with the most simple
example from the web2py's SOAP service document is good enough.
The usage looks something like this:
try {
String url = http://; + ServerIP + / +
AppID + /default/call/soap;
// Create SOAP Connection
SOAPConnectionFactory soapConnectionFactory =
SOAPConnectionFactory.newInstance();
SOAPConnection soapConnection = soapConnectionFactory.
createConnection();
// Send SOAP Message to SOAP Server
SOAPMessage soapResponse = soapConnection.call(
createSOAPRequest(tdName, tdiskStatus),
url);
// Process the SOAP Response
printSOAPResponse(soapResponse);
soapConnection.close();
} catch (SOAPException ex) {
System.out.println(main: soap + ex.getMessage());
Logger.getLogger(LogTDSClient.class.getName()).log(Level.SEVERE,
null, ex);
} catch (UnsupportedOperationException ex) {
System.out.println(main: uns + ex.getMessage());
Logger.getLogger(LogTDSClient.class.getName()).log(Level.SEVERE,
null, ex);
} catch (Exception ex) {
System.out.println(main: oops + ex.getMessage());
Logger.getLogger(LogTDSClient.class.getName()).log(Level.SEVERE,
null, ex);
}
and the interesting part:
private static SOAPMessage createSOAPRequest(String tName, String
tStatus) throws Exception {
MessageFactory messageFactory = MessageFactory.newInstance();
SOAPMessage soapMessage = messageFactory.createMessage();
SOAPPart soapPart = soapMessage.getSOAPPart();
String serverURI = http://your-url:here;;
String authHack = ZGxv-usual-base64stuff;
// SOAP Envelope
SOAPEnvelope envelope = soapPart.getEnvelope();
// your-url:here corresponds to serverIP above
envelope.addNamespaceDeclaration(ns0,
http://your-url:here/AppID/default/call/soap;);
// SOAP Body
SOAPBody soapBody = envelope.getBody();
SOAPElement soapBodyElem = soapBody.addChildElement(YourSoapFunc
, ns0);
SOAPElement soapBodyElem1 = soapBodyElem.addChildElement(
YourSoapParam);
soapBodyElem1.addTextNode(YourSoapParamString);
MimeHeaders headers = soapMessage.getMimeHeaders();
headers.addHeader(SOAPACTION, );
headers.addHeader(Authorization, Basic + authHack);
Iterator outHeaders = headers.getAllHeaders();
while (false outHeaders.hasNext()) {
MimeHeader outH = (MimeHeader) outHeaders.next();
// System.out.println(outH.getName() + -- +outH.getValue());
}
soapMessage.saveChanges();
/* Print the request message */
if (Boolean.FALSE) {
System.out.print(Request SOAP Message = );
soapMessage.writeTo(System.out);
System.out.println();
}
return soapMessage;
}
It's pretty simple. You might need to tweak the return value handling,
because that part wasn't very important to my usage.
(Grrr -- the GG message editor gets funky when I put in a code block, at
least in this browser.)
Good luck.
/dps
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Java Client for web2py SOAP web service
I can make the client work if using python suds and pysimplesoap. But when
I use JAVA-WS client, I always got the error:
javax.xml.ws.WebServiceException: Failed to access the WSDL at:
https://aa.bb.cc.dd//default/call/soap?WSDL. It failed with:
Got java.security.cert.CertificateException: No subject alternative names
present while opening stream from
https://aa.bb.cc.dd//default/call/soap?WSDL.
Here aa.bb.cc.dd is a public ip address. The web2py application is host on
amazon EC2.
Thanks!
On Monday, November 24, 2014 3:24:25 PM UTC-5, Pengfei Yu wrote:
Hi Dave,
Thanks very much for your reply! Could you give a more detailed example
for an implementation for Java client? A block of java codes from your Java
client talking to web2py service will be perfect. Even with the most simple
example from the web2py's SOAP service document is good enough.
Thanks!
On Friday, November 21, 2014 6:16:45 PM UTC-5, Dave S wrote:
On Thursday, November 20, 2014 6:49:18 AM UTC-8, Pengfei Yu wrote:
Hi,
I am using the SOAP service provided by web2py to create a simple web
service for my web application. The document provide a example of using
pysimplesoap to consume the created SOAP web service. The code is like
below:
from gluon.contrib.pysimplesoap.client import SoapClient
client =
SoapClient(wsdl=http://localhost:8000/app/default/call/soap?WSDL;)
print client.MyAdd(a=1,b=2)
{'result': 3}
My question is for this simple web service, can we use other languages
like JAVA to consume it in the client side. Does the simple web service
allow JAVA clients like JAX-WS or JAVA Axis to access it?
My cooperator want me to create a web service for my application and
they are using JAVA.
I have a Java client talking to my Web2Py service.
My imports include javax.xml.soap.xml, and i instantiate using
SOAPConnectionFactory.
I also have a Python client that talks to the same service using suds
rather than pysimplesoap, so I would expect any typical client library to
work. Mariano seems to have a very good library for us to leverage.
/dps
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Java Client for web2py SOAP web service
Hi Dave,
Thanks very much for your reply! Could you give a more detailed example for
an implementation for Java client? A block of java codes from your Java
client talking to web2py service will be perfect. Even with the most simple
example from the web2py's SOAP service document is good enough.
Thanks!
On Friday, November 21, 2014 6:16:45 PM UTC-5, Dave S wrote:
On Thursday, November 20, 2014 6:49:18 AM UTC-8, Pengfei Yu wrote:
Hi,
I am using the SOAP service provided by web2py to create a simple web
service for my web application. The document provide a example of using
pysimplesoap to consume the created SOAP web service. The code is like
below:
from gluon.contrib.pysimplesoap.client import SoapClient
client =
SoapClient(wsdl=http://localhost:8000/app/default/call/soap?WSDL;)
print client.MyAdd(a=1,b=2)
{'result': 3}
My question is for this simple web service, can we use other languages
like JAVA to consume it in the client side. Does the simple web service
allow JAVA clients like JAX-WS or JAVA Axis to access it?
My cooperator want me to create a web service for my application and they
are using JAVA.
I have a Java client talking to my Web2Py service.
My imports include javax.xml.soap.xml, and i instantiate using
SOAPConnectionFactory.
I also have a Python client that talks to the same service using suds
rather than pysimplesoap, so I would expect any typical client library to
work. Mariano seems to have a very good library for us to leverage.
/dps
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Java Client for web2py SOAP web service
Hi,
I am using the SOAP service provided by web2py to create a simple web
service for my web application. The document provide a example of using
pysimplesoap to consume the created SOAP web service. The code is like
below:
from gluon.contrib.pysimplesoap.client import SoapClient
client = SoapClient(wsdl=http://localhost:8000/app/default/call/soap?WSDL;)
print client.MyAdd(a=1,b=2)
{'result': 3}
My question is for this simple web service, can we use other languages like
JAVA to consume it in the client side. Does the simple web service allow
JAVA clients like JAX-WS or JAVA Axis to access it?
My cooperator want me to create a web service for my application and they
are using JAVA.
Thanks!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler always QUEUED and no record in scheduler_worker and scheduler_run data table
Sorry, There is a typo in the previous question. In this sentence In the data table scheduler_worker and scheduler_run I want to delete both entries, what I want to say is scheduler_task and scheduler_run tables. In my design, I linked a project with a task by adding an field called processID for each project which is identical to uuid of corresponding task. So that the project table and scheduler_task table could be linked. You are right that each time I delete a record in scheduler_task, one linked record in scheduler_run need to be deleted as well (assuming that the task is completed). So I currently I used this script to delete a record in project and scheduler_task and scheduler_run: def delete(): processID = request.args(0) project = db.project(db.project.processID==processID) db(db.project.processID==processID).delete() task = db.scheduler_task(db.scheduler_task.uuid==processID) db(db.scheduler_run.task_id==task.id).delete() db(db.scheduler_task.uuid==processID).delete() On Saturday, October 4, 2014 2:51:42 PM UTC-4, Niphlod wrote: On Friday, October 3, 2014 8:19:46 PM UTC+2, Pengfei Yu wrote: Thanks a lot for your detailed explanation for the difference between Celery and scheduler! I think currently scheduler is enough for our application, and it is easier to implement (thanks to great work of your team). May I have one more question. If I want to allow user to delete one of his projects, which means the task associated with the project will also be deleted, is there a simple command in scheduler to do that. In the data table scheduler_worker and scheduler_run I want to delete both entries. Should I use db(db.scheduler_*.uuid==an_id).delete() for both of them, or there is a single command like scheduler.delete_task(uuid) to do that. I am not sure if scheduler.terminate(), scheduler.stop_task() or scheduler.kill can do that. I think you need to make a tour on w2p_scheduler_test https://github.com/niphlod/w2p_scheduler_tests beforehand. It seems that some basic concept on what is the scheduler API is somehow obscure to you. The scheduler is a way to let some task be processed outside web2py. How you (and anyone else) manage the correlation between tasks sent to the scheduler and their own app (e.g., your requirement for them to be assigned to a project) is entirely up to your application's code. The data in scheduler_worker shouldn't matter to you: it's the table where workers coordinate among themselves and carries some statistics around. scheduler_task is the table that holds all the tasks you sent (and where you can check their statuses). scheduler_run is the table where return data (if needed) is stored. That being said, if you delete a task, its return data (if it's there), is deleted, because there is a foreign key that links the two tables (to be even more plain in explanations, there's no way you can delete a scheduler_task row without deleting ALSO the corresponding records on scheduler_run). Ultimately, the kill() and terminate() methods are to there manage worker processes (i.e. the things that are processing tasks), not tasks directly. stop_task() instead is a method to manage a possibly RUNNING task (i.e. something that you clearly can't dequeue, because it's already being processed), asking the corresponding worker to stop processing it (so it can be free to process other tasks). calling stop_task on a task that is not RUNNING will result in that task never being processed (it will be marked as STOPPED and nobody will pick that up) -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler always QUEUED and no record in scheduler_worker and scheduler_run data table
Thanks a lot for your detailed explanation for the difference between Celery and scheduler! I think currently scheduler is enough for our application, and it is easier to implement (thanks to great work of your team). May I have one more question. If I want to allow user to delete one of his projects, which means the task associated with the project will also be deleted, is there a simple command in scheduler to do that. In the data table scheduler_worker and scheduler_run I want to delete both entries. Should I use db(db.scheduler_*.uuid==an_id).delete() for both of them, or there is a single command like scheduler.delete_task(uuid) to do that. I am not sure if scheduler.terminate(), scheduler.stop_task() or scheduler.kill can do that. On Friday, October 3, 2014 2:23:22 AM UTC-4, Niphlod wrote: On Wednesday, October 1, 2014 4:34:08 PM UTC+2, Pengfei Yu wrote: So that means all the communications between different machines are from database management system, and has little thing to do with the scheduler? Sorry, I am new to this field and the question may be stupid. all the communications between any process sitting on any system just needs to access the database you point your scheduler instance to. And I see Celery has features to support distributed computing ( http://stackoverflow.com/questions/23916413/celery-parallel-distributed-task-with-multiprocessing), Are those features also included in Scheduler? same exact thing, two different concepts in mind: web2py's scheduler assumes that the time it takes you to send a task is neglegible in regard of the actual time the task will take to be processed, while celery use a software stack that tries hard to minimize that timespan. Also, consider that the whole Celery team is focused on the creation of a task processor, while web2py's is not (just sayin') In real world terms, this means that if your goal is to process 1 trillion task per minute, Celery (and a big freaky farm of servers) is the right tool for the job. Don't even start with web2py's scheduler with those numbers. web2py's scheduler performances are hardly dependant on the underlying db (since it's where all of the communication and coordination for all the activities takes place). Scheduler's usually runs out (on commodity hardware) at 2000 tasks per minute. Still, it's usually enough for a vast majority of web applications. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler always QUEUED and no record in scheduler_worker and scheduler_run data table
So that means all the communications between different machines are from database management system, and has little thing to do with the scheduler? Sorry, I am new to this field and the question may be stupid. And I see Celery has features to support distributed computing (http://stackoverflow.com/questions/23916413/celery-parallel-distributed-task-with-multiprocessing), Are those features also included in Scheduler? Thanks! On Wednesday, October 1, 2014 3:16:57 AM UTC-4, Niphlod wrote: On Tuesday, September 30, 2014 3:42:41 PM UTC+2, Pengfei Yu wrote: Hi Simone, Thanks for your reply. I used the solution from the web2py document site: http://www.web2py.com/books/default/chapter/29/13/deployment-recipes, in section Start the scheduler as a Linux service (upstart). After I started it, the status check (sudo status web2py-scheduler) shows that it is already started: ubuntu@ip-172-31-41-2:~$ sudo status web2py-scheduler web2py-scheduler start/running, process 16262 Interesting thing is that when I restart web2py-scheduler again, it works out. The previous worker didn't show up in the db_scheduler_worker table but the current one shows up. well, solutions are ready to use but to check nonetheless ^_^' May I have another question? We are planning to use the power of distributed system cluster with multiple EC2 instances for this project. May I ask if the tasks in the scheduler just use different thread in a CPU or it can also be designed to use different cores or nodes with multiple processors? (From this post: https://groups.google.com/forum/#!topic/web2py/WlhvelFu8-Q, it seems the scheduler can be adapted for multiple cores). And if that is the case, is there any special design on web2py scheduler itself for the communication between master instance and others? each scheduler instance is a process. there's no piece of code in earth (in any programming language) that'll make a program go on a specific processor: that's something only the underlying operating system manages. The only communication needed is the connection to the db. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: Scheduler always QUEUED and no record in scheduler_worker and scheduler_run data table
Hi Simone,
Thanks for your reply. I used the solution from the web2py document site:
http://www.web2py.com/books/default/chapter/29/13/deployment-recipes, in
section Start the scheduler as a Linux service (upstart). After I started
it, the status check (sudo status web2py-scheduler) shows that it is
already started:
ubuntu@ip-172-31-41-2:~$ sudo status web2py-scheduler
web2py-scheduler start/running, process 16262
Interesting thing is that when I restart web2py-scheduler again, it works
out. The previous worker didn't show up in the db_scheduler_worker table
but the current one shows up.
May I have another question? We are planning to use the power of
distributed system cluster with multiple EC2 instances for this project.
May I ask if the tasks in the scheduler just use different thread in a CPU
or it can also be designed to use different cores or nodes with multiple
processors? (From this post:
https://groups.google.com/forum/#!topic/web2py/WlhvelFu8-Q, it seems the
scheduler can be adapted for multiple cores). And if that is the case, is
there any special design on web2py scheduler itself for the communication
between master instance and others?
Thanks a lot!
On Tuesday, September 30, 2014 3:04:07 AM UTC-4, Niphlod wrote:
if there are no records in the scheduler_worker, the scheduler is not
running. Please check your permanent daemon solution to see if it's
running and what it logs.
On Monday, September 29, 2014 4:37:53 PM UTC+2, Pengfei Yu wrote:
Hi,
I have a web application hosted on AWS EC2 with Apache2. I plan to use
web2py scheduler for some heavy tasks with user input files. Now I only use
some small tasks like wc -c to test the functions of scheduler. It seems
that the submitted tasks are always QUEUED and the status never changes.
After searching the google group, I added db.commit() right after
scheduler.queue_task(), but it still cannot work.
I've already Installed the scheduler as a permanent daemon on Linux and
started web2py-scheduler using sudo start web2py-scheduler.
Here is part of my codes in default.py:
def index():
response.flash = T(Welcome to web2py!)
form=SQLFORM(db.project,fields=['name',
'Genome',File_type,InputFile])
if request.vars.InputFile!=None and
request.vars.InputFile.filename.split(.)[-1]!=request.vars.File_type:
response.flash = 'The input file must match file type, please
fill again'
return dict(form=form)
if request.vars.InputFile!=None:
form.vars.processID=0
if form.process().accepted:
response.flash = 'form accepted'
filepath = form.vars.InputFile
command = wc -c
/home/www-data/web2py/applications/BamToVcf/uploads/+filepath
p = scheduler.queue_task('run',[command])
db.commit()
row = db(db.project.InputFile==filepath).select().first()
row.update_record(processID = p.uuid)
redirect(URL(progress,args=[p.uuid]))
elif form.errors:
response.flash = 'form has errors'
else:
response.flash = please fill out the form
return dict(form=form)
def progress():
processID = request.args(0)
project = db.project(db.project.processID==processID)
status=scheduler.task_status(processID,output=True)
start_time = utc_to_local(status.scheduler_task.start_time)
return dict(project=project,p=status,start_time=start_time)
def list():
rows = db(db.project.id0).select()
return dict(rows=rows)
And the code in scheduler.py in Models folder:
from gluon.scheduler import Scheduler
import subprocess
from collections import OrderedDict
import os
def run(command):
p = subprocess.Popen(command,
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
startTime = time.time()
stdout, stderr = p.communicate()
endTime = time.time()
return (command, p.returncode, stdout, stderr, round(endTime -
startTime))
def demo2(a,b):
return a+b
scheduler = Scheduler(db,dict(run=run,demo2=demo2))
From the database page (~/%app/appadmin/index), I can see the records
of tasks in db.scheduler_task table but no record in db.scheduler_worker
and db.scheduler_run table.
Could any one help me figure out the problem?
Thanks very much!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[web2py] Scheduler always QUEUED and no record in scheduler_worker and scheduler_run data table
Hi,
I have a web application hosted on AWS EC2 with Apache2. I plan to use
web2py scheduler for some heavy tasks with user input files. Now I only use
some small tasks like wc -c to test the functions of scheduler. It seems
that the submitted tasks are always QUEUED and the status never changes.
After searching the google group, I added db.commit() right after
scheduler.queue_task(), but it still cannot work.
I've already Installed the scheduler as a permanent daemon on Linux and
started web2py-scheduler using sudo start web2py-scheduler.
Here is part of my codes in default.py:
def index():
response.flash = T(Welcome to web2py!)
form=SQLFORM(db.project,fields=['name',
'Genome',File_type,InputFile])
if request.vars.InputFile!=None and
request.vars.InputFile.filename.split(.)[-1]!=request.vars.File_type:
response.flash = 'The input file must match file type, please fill
again'
return dict(form=form)
if request.vars.InputFile!=None:
form.vars.processID=0
if form.process().accepted:
response.flash = 'form accepted'
filepath = form.vars.InputFile
command = wc -c
/home/www-data/web2py/applications/BamToVcf/uploads/+filepath
p = scheduler.queue_task('run',[command])
db.commit()
row = db(db.project.InputFile==filepath).select().first()
row.update_record(processID = p.uuid)
redirect(URL(progress,args=[p.uuid]))
elif form.errors:
response.flash = 'form has errors'
else:
response.flash = please fill out the form
return dict(form=form)
def progress():
processID = request.args(0)
project = db.project(db.project.processID==processID)
status=scheduler.task_status(processID,output=True)
start_time = utc_to_local(status.scheduler_task.start_time)
return dict(project=project,p=status,start_time=start_time)
def list():
rows = db(db.project.id0).select()
return dict(rows=rows)
And the code in scheduler.py in Models folder:
from gluon.scheduler import Scheduler
import subprocess
from collections import OrderedDict
import os
def run(command):
p = subprocess.Popen(command,
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
startTime = time.time()
stdout, stderr = p.communicate()
endTime = time.time()
return (command, p.returncode, stdout, stderr, round(endTime -
startTime))
def demo2(a,b):
return a+b
scheduler = Scheduler(db,dict(run=run,demo2=demo2))
From the database page (~/%app/appadmin/index), I can see the records of
tasks in db.scheduler_task table but no record in db.scheduler_worker and
db.scheduler_run table.
Could any one help me figure out the problem?
Thanks very much!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
