[web2py] Re: Call functions from other applications without network requests

[Anthony]

> If I understand right, the second one generates an HTTP request.  That 
> means the data that is fetched will be serialized and deserialized.  In 
> some cases this data is sizable, so I want to avoid that overhead.  There 
> will likely also be some overhead associated with actually 
> sending/receiving the network request; this is probably smaller but I'd 
> still rather avoid it.  As I understand it, these operations will also 
> happen with an RPC call.  That's why I keep saying that I don't want an 
> HTTP request.  I want everything to happen totally internally, within the 
> same request, same process, same everything, nothing going over the wire.
>

Correct, there would be some overhead due to the HTTP request.
 

> Because work_service is defined as a JSON-RPC service, the proper way to 
>> call it would be via an RPC call, as explained here: 
>> http://web2py.com/books/default/chapter/29/10/services#Accessing-JSONRPC-services-from-web2py
>>  
>> .
>>  
>> Otherwise, there is not much point in making it an RPC service.
>>
>  
> It's not an RPC service per se.  The decorator is @service.json, not 
> @service.jsonrpc.
>

Oh, right, sorry about that. The same reasoning applies, though -- you 
would simply make an HTTP request rather than use an RPC client.
 

> That is also why it's not purely a matter of "common functionality that is 
> shared across apps".  Rather, there is functionality that I want to be 
> accessible BOTH as a "real" URL endpoint AND internally between apps --- 
> but without the request overhead in the latter case.
>

I suppose it would help to understand your real world use case. It's not 
clear what the problem is with using modules. It might also help to know 
how you would handle this in other frameworks.
 

>   So I do want a controller that exposes the function.  (I could of course 
> write a "dummy" controller function that just calls a "worker" function in 
> a module, but that is essentially what Service already gives me.)
>

Not exactly. The functions exposed by Service are not accessible as HTTP 
request endpoints on their own.

Note, if you have a lot of functions in a module that all need to be 
exposed as controller endpoints, you could route to them via request.args. 
For example, in /modules/mycontroller.py:

from gluon import current

def myfunc():
return dict(message=current.request.vars.message)

And in /controllers/default.py:

import mycontroller

def index():
return getattr(mycontroller, request.args(0))()

Then access /myapp/default/index/myfunc to get the output of myfunc (you 
can use the router to hide "default" and "index", so the URL can be 
re-written to /myapp/myfunc). You can put as many functions as you want in 
mycontroller.py, and they can be accessed from any app.

Here is an example of this approach: 
https://github.com/TechEmpower/FrameworkBenchmarks/tree/2623eae7135205c8ca4da27d44ea1ccda6c19a56/frameworks/Python/web2py/app/standard.

Anthony

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Brendan Barnwell]

On Saturday, February 18, 2017 at 6:00:23 AM UTC-8, Anthony wrote:
>
> Do you have a use case that cannot easily be accommodated by that pattern? 
>> If so, what do you imagine would be the API for doing what you want to do, 
>> and would it look much different from simply making an HTTP call (or 
>> possibly an RPC call) to the other app (which you can already do)?
>>
>
> And just to be clear, if you are OK with this:
>
> def order_plain():
> othermod = gluon.shell.exec_environment(
> 'applications/test_env/controllers/db_worker.py')
> return "Hello", othermod.work_plain()
>
> then why not this:
>
> def order_plain():
> work_plain_response = gluon.tools.fetch(URL('db_worker', 'work_plain', 
> host='localhost'))
> return "Hello %s" % work_plain_response
>

If I understand right, the second one generates an HTTP request.  That 
means the data that is fetched will be serialized and deserialized.  In 
some cases this data is sizable, so I want to avoid that overhead.  There 
will likely also be some overhead associated with actually 
sending/receiving the network request; this is probably smaller but I'd 
still rather avoid it.  As I understand it, these operations will also 
happen with an RPC call.  That's why I keep saying that I don't want an 
HTTP request.  I want everything to happen totally internally, within the 
same request, same process, same everything, nothing going over the wire.

In your other message you said:

The workflow of an HTTP request involves things like setting up the 
> session, and some of the code (particularly controller code) may depend on 
> attributes of the request object. If you want to make what amounts to an 
> internal request to another app or controller, it will not be clear what 
> elements of the current request to retain (e.g., should the current 
> session, request headers, query string, etc. be replicated in the second 
> call?).
>

Yeah, that is true.  To my mind, though, the point of using something like 
Service is to constrain the function's interface so that the situation you 
describe won't arise.  Instead, the function is supposed to depend only on 
the arguments you pass it.  However, I can see that it is difficult for 
web2py to support this use case, because there is no way to enforce the 
restriction that nothing in the code path (i.e., nothing in the model or 
controller code) directly accesses the request object.  In some sense, even 
my function does not really just depend on its arguments, because it still 
depends on the global "db" object.  What I was hoping is that there is a 
way to get stuff like "db" (i.e., global objects I created in my model 
code) without the need to have objects like "request" (i.e., global objects 
built into web2py).  But I see your point: the normal web2py workflow 
places no constraints on how code may access request/response/session/etc.  
Since these objects are "allowed" to be directly accessed (and even 
mutated) from anywhere in user code, trying to call code without them would 
probably fail more often than not.

(I mostly included work_plain there just to be able to verify that the db 
object wasn't being created.  My real use case is more like work_service.)

Because work_service is defined as a JSON-RPC service, the proper way to 
> call it would be via an RPC call, as explained here: 
> http://web2py.com/books/default/chapter/29/10/services#Accessing-JSONRPC-services-from-web2py
>  
> .
>  
> Otherwise, there is not much point in making it an RPC service.
>
 
It's not an RPC service per se.  The decorator is @service.json, not 
@service.jsonrpc.  The goal of using Service here is what I mentioned 
above: to allow the function's API to be its argument signature, rather 
than allowing it to grab arbitrary stuff from the request object.  The idea 
is that sometimes I do want to actually visit the URL in a browser (or 
retrieve it programmatically), and in that case of course a real request 
must be generated; but just in case the "request" is actually an internal 
call, I would rather have a "shortcut" that doesn't do anything that has to 
do with going over the wire.

That is also why it's not purely a matter of "common functionality that is 
shared across apps".  Rather, there is functionality that I want to be 
accessible BOTH as a "real" URL endpoint AND internally between apps --- 
but without the request overhead in the latter case.  So I do want a 
controller that exposes the function.  (I could of course write a "dummy" 
controller function that just calls a "worker" function in a module, but 
that is essentially what Service already gives me.)

Anyway, thanks for your help on this matter, I'll look at modularizing my 
db-creation code a bit so that I can more easily create the db in each 
module that needs 

[web2py] Re: Call functions from other applications without network requests

[Anthony]

It may also be worth noting that you can get the output of another 
controller function within the *same *application via LOAD(..., ajax=False) 
(note, ajax=False is actually the default). This, however, will also 
execute the called function's view and return the full rendered output. In 
this case, the execution environment of the current request is re-used 
(i.e., it does not re-execute the models but simply runs the requested 
function in the current environment), which is why this does not work 
across applications. So, taking your example, you could do:

def order_plain():
work_plain_response = LOAD('db_worker', 'work_plain')
return "Hello %s" % work_plain_response

Anthony

On Saturday, February 18, 2017 at 9:00:23 AM UTC-5, Anthony wrote:
>
> Do you have a use case that cannot easily be accommodated by that pattern? 
>> If so, what do you imagine would be the API for doing what you want to do, 
>> and would it look much different from simply making an HTTP call (or 
>> possibly an RPC call) to the other app (which you can already do)?
>>
>
> And just to be clear, if you are OK with this:
>
> def order_plain():
> othermod = gluon.shell.exec_environment(
> 'applications/test_env/controllers/db_worker.py')
> return "Hello", othermod.work_plain()
>
> then why not this:
>
> def order_plain():
> work_plain_response = gluon.tools.fetch(URL('db_worker', 'work_plain', 
> host='localhost'))
> return "Hello %s" % work_plain_response
>
> Anthony
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Anthony]

>
> Do you have a use case that cannot easily be accommodated by that pattern? 
> If so, what do you imagine would be the API for doing what you want to do, 
> and would it look much different from simply making an HTTP call (or 
> possibly an RPC call) to the other app (which you can already do)?
>

And just to be clear, if you are OK with this:

def order_plain():
othermod = gluon.shell.exec_environment(
'applications/test_env/controllers/db_worker.py')
return "Hello", othermod.work_plain()

then why not this:

def order_plain():
work_plain_response = gluon.tools.fetch(URL('db_worker', 'work_plain', 
host='localhost'))
return "Hello %s" % work_plain_response

Anthony

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Anthony]

> If I visit /test_env/db_worker/call/json/work_service?a=1=2, I get the 
> right result.  What I want is to be able to visit 
> /test_env/boss/order_service and get the exact same result.  Right now, I 
> get "name 'service' is not defined" (because db.py is not run, and 
> "service" is defined there).
>

Because work_service is defined as JSON-RPC service, the proper way to call 
it would be via an RPC call, as explained here: 
http://web2py.com/books/default/chapter/29/10/services#Accessing-JSONRPC-services-from-web2py.
 
Otherwise, there is not much point in making it an RPC service.
 

> Now, as near as I can tell, everything I've done above in db_worker.py is 
> straight out of the book as basic web2py stuff.  But what you seem to be 
> saying is that, if I want to make "order" work, I have to significantly 
> change the way I've written db.py and worker.py.  Namely, I have to abandon 
> the idea of the normal execution framework whereby model files are 
> automatically executed and dump their contents into the namespace of 
> controllers, and instead I have to factor out the db-creation code into a 
> separate module, and then, instead of having it automatically executed and 
> provided to the db_worker controller, I have to manually import and run 
> that db-creation code from within db_worker.py.  Is that correct?
>

You make it sound more complicated than it is. If you need to share both a 
database and model definitions across applications, you can simply take the 
code you would otherwise have put in a model file and instead put it in a 
function in a module. Then in each app that needs it, import the function 
and run it. This involves maybe a couple extra lines of code in each app.

Note, web2py already employs this approach to provide some of its 
functionality. Consider Auth and Scheduler -- both of those classes take a 
db object and define a number of models. Yet they are defined in modules 
and must be imported and instantiated in any given application that wants 
to use them. This is the standard mechanism for sharing code in Python.
 

> If so, I understand what you're saying, but I guess it seems a bit 
> perverse.  Web2py has all this nice magic for automatically deciding which 
> model files to run, running them, and automatically providing the names 
> they define to the controller files.  It's great and makes things very 
> convenient and reduces boilerplate.  But that all has to go out the window 
> for one controller to access another (even within the same app)?
>

The workflow of an HTTP request involves things like setting up the 
session, and some of the code (particularly controller code) may depend on 
attributes of the request object. If you want to make what amounts to an 
internal HTTP request to another app or controller, it will not be clear 
what elements of the current request to retain (e.g., should the current 
session, request headers, query string, etc. be replicated in the second 
call?). Note, you can achieve this to some extent by calling 
gluon.shell.env and passing in some or all of the current request 
attributes via the extra_request parameter, but things can get messy very 
quickly. Should everything happen in the same database transaction? What if 
there is a redirect or HTTP exception in the second app? You will generally 
be better off isolating full HTTP requests to a single app.
 

> Is there no way to say "do everything just as you would if a request came 
> in for it -- that is, run the model files and the controller file -- but 
> just don't actually create a request or call any of the controller 
> functions yet, just return me an object representing the controller file"?
>

I wonder how often you really need to do something like this. If there is 
some common set of functionality that needs to be shared across apps, it 
probably does not have to be in the form of a controller function, but 
could instead be a more general function that can be used as part of a 
controller. Again, consider how the Auth functionality works -- it does not 
provide a set of ready-made controller functions but instead provides a 
mechanism to (a) define a common set of database tables and (b) expose 
methods that can be called from within controllers where needed. Do you 
have a use case that cannot easily be accommodated by that pattern? If so, 
what do you imagine would be the API for doing what you want to do, and 
would it look much different from simply making an HTTP call (or possibly 
an RPC call) to the other app (which you can already do)?

Anthony

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For 

[web2py] Re: Call functions from other applications without network requests

[Brendan Barnwell]

> The simplest approach is to put shared functionality into modules. Of 
> course, if you don't know in what context a given bit of code will run, you 
> can easily check whether particular tables have been defined and then call 
> the table defining code if necessary.
>
> Maybe it would help to show some actual code from two apps that you would 
> like to share.
>

Yeah, you're right, it is getting a bit abstract.  I made a simple 
example.  This is with only one app (with two controllers) but it seems to 
illustrate the same kind of problem I'm having with multiple apps, so maybe 
we can start with this.  I created a sample app called test_env.  
Everything in db.py is left as the default, except I uncommented 
request.requires_https().  I then made two controller files:

# db_worker.py
def call():
return service()

@service.json
def work_service(a, b=2):
return a, b, str(db)

def work_plain():
return str(db)

# boss.py
import gluon.shell

def order_service():
othermod = gluon.shell.exec_environment(
'applications/test_env/controllers/db_worker.py')
return "Hello", othermod.work(a=1, b=2)

def order_plain():
othermod = gluon.shell.exec_environment(
'applications/test_env/controllers/db_worker.py')
return "Hello", othermod.work_plain()

I just used "print the db object" as a dummy "do something with the db" 
usage.

If I visit /test_env/db_worker/call/json/work_service?a=1=2, I get the 
right result.  What I want is to be able to visit 
/test_env/boss/order_service and get the exact same result.  Right now, I 
get "name 'service' is not defined" (because db.py is not run, and 
"service" is defined there).  Likewise, work_plain works, but order_plain 
gives the same error.  If I comment out work_service (to bypass the "name 
service not found" error) and try order_plain, I now get "name 'db' is not 
defined" (because "db" is also defined in db.py, which is still not run).

Now, as near as I can tell, everything I've done above in db_worker.py is 
straight out of the book as basic web2py stuff.  But what you seem to be 
saying is that, if I want to make "order" work, I have to significantly 
change the way I've written db.py and worker.py.  Namely, I have to abandon 
the idea of the normal execution framework whereby model files are 
automatically executed and dump their contents into the namespace of 
controllers, and instead I have to factor out the db-creation code into a 
separate module, and then, instead of having it automatically executed and 
provided to the db_worker controller, I have to manually import and run 
that db-creation code from within db_worker.py.  Is that correct?

If so, I understand what you're saying, but I guess it seems a bit 
perverse.  Web2py has all this nice magic for automatically deciding which 
model files to run, running them, and automatically providing the names 
they define to the controller files.  It's great and makes things very 
convenient and reduces boilerplate.  But that all has to go out the window 
for one controller to access another (even within the same app)?  Is there 
no way to say "do everything just as you would if a request came in for it 
-- that is, run the model files and the controller file -- but just don't 
actually create a request or call any of the controller functions yet, just 
return me an object representing the controller file"?

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Anthony]

>
> What I don't like about this is that it means the DB-setup code is run 
>>> multiple times on a "normal" request.  That is, if I have code to set up 
>>> the DB in my model file, and I visit my db-accessing controller, it runs 
>>> the code in the model file to set up the DB, then runs the code again when 
>>> it runs the controller file.  I imagine this is not a huge penalty but it 
>>> is annoying.
>>>
>>
>> So, then don't do it that way. You have two options. First, if you have a 
>> controller that doesn't need any of the model definitions from your 
>> application's model files, then you should probably re-architect your 
>> application so only the models that are needed get defined when they are 
>> needed (either using the conditional models functionality or by defining 
>> all models in modules that are imported where needed).
>>
>
> Not sure what you mean there about the models.  What you describe is the 
> opposite of my situation.  My controller does need the models.  That is 
> exactly the problem, because I want to call that controller from another 
> controller, but to do so I need the model files to be run.
>

You said you were running the db setup code twice. Since the second run 
overwrites the first, by definition you must not need the first, so just 
avoid running it the first time.
 

> The other option is simply to avoid re-running the DB setup code and 
>> instead use the already defined db object, passing it to the module code 
>> from the other app:
>>
>> app1/models/db.py:
>>
>> db = DAL(...)
>>
>> app2/modules/db_definer.py:
>>
>> def define_tables(db):
>> db.define_table('my_app2_table', ...)
>>
>> app1/controllers/default.py:
>>
>> from app2.modules.db_definer import define_tables
>>
>> def myfunc():
>> define_tables(db)
>> ...
>>
>> The above assumes both apps are using the same database, but if they are 
>> not, then there is no "re-running the DB setup code" anyway, as you would 
>> be dealing with two different databases, which would each need their own 
>> setup code.
>>  
>>
>
> What already-defined DB object?
>

You weren't quite clear about what you were doing. You showed the following 
code:

from applications.otherapp.modules.db_definer
db = db_definer.define_db()

and complained that the code to create the "db" object was being run twice 
(presumably once in this app's model files, and once in the db_definer 
module from otherapp). My suggestion is that if both "db" objects are for 
the same database, then just create the "db" object once (i.e., in this 
app's model files) and pass it to any module code that needs to define 
specific tables (i.e., don't create the "db" object in the module).
 

>   If I use exec_environment, model files aren't run, so there is no db 
> object.
>

My assumption is we are not using exec_environment here. I don't think that 
will be a fruitful approach.
 

> Anyway, your response makes me think maybe I haven't explained my 
> situation correctly.  What I have this: controllerA.py and controllerB both 
> needs the model in modelA.py  controllerB wants to call a function in 
> controllerA.  What I want is a way, from within controller B, to say "run 
> controllerA, making sure all the models it needs are available to it, and 
> then give me controllerA as a module object (or something equivalent), so 
> that I can call controllerA.some_function() and get the results".  All 
> without any additional network requests.  (In fact, some_function is a 
> service, so I will pass arguments to it.).
>
 

> If you need to share code across applications, I suggest you put that code 
>> into modules and then import in whatever applications need the 
>> functionality. Of course, you can also make internal HTTP requests from one 
>> application to another (or use an RPC mechanism).
>>
>
> How would such code access the DB?  By putting the define_tables-type code 
> at the module top-level?  Or by putting it in a function and having every 
> db-accessing function call "db = get_DB()" at the beginning?
>

Those are not the only options. First, note that you can separate the 
creation of the DAL object and the defining of its tables. You could have 
one function in a module define the DAL object for a given database and 
call that function from the model file of any given app. You can then have 
different functions for defining various models, and those functions can be 
called from anywhere -- a model, the top level of a controller, within a 
controller function, or even another module.
 

>   That seems similar to what I'm doing, except that the code is in a 
> controller instead of a module.  It seems the bottom line is that there is 
> no way for a controller to internally specify a "dependency" on particular 
> models.
>

If you create functions in modules to define the models, you can specify 
the dependency by importing the module from the controller.
 

>   The normal web2py mechanism will run the right model files on a regular 
> 

[web2py] Re: Call functions from other applications without network requests

[Brendan Barnwell]

On Thursday, February 16, 2017 at 6:39:37 AM UTC-8, Anthony wrote:
>
> In your example above, it's one model defining tables from another model.  
>> But in my case, what I have is a controller function that needs access to 
>> the db, which is defined in the model file.  But I found the hard way (and 
>> later also from an old thread: 
>> https://groups.google.com/forum/#!topic/web2py-developers/5TPI9oYOZHM ) 
>> that exec_environment does not load models.  So this puts me kind of back 
>> to where I was.
>>
>> The simplest solution I found was to put, in the controller file, code 
>> that looks like this:
>>
>> from applications.otherapp.modules.db_definer
>> db = db_definer.define_db()
>>
>> Then every time my controller is run, it will redefine the DB, using the 
>> code set up in the other app's module to do this in a consistent way (along 
>> the lines you suggested).
>>
>> What I don't like about this is that it means the DB-setup code is run 
>> multiple times on a "normal" request.  That is, if I have code to set up 
>> the DB in my model file, and I visit my db-accessing controller, it runs 
>> the code in the model file to set up the DB, then runs the code again when 
>> it runs the controller file.  I imagine this is not a huge penalty but it 
>> is annoying.
>>
>
> So, then don't do it that way. You have two options. First, if you have a 
> controller that doesn't need any of the model definitions from your 
> application's model files, then you should probably re-architect your 
> application so only the models that are needed get defined when they are 
> needed (either using the conditional models functionality or by defining 
> all models in modules that are imported where needed).
>

Thanks for your quick reply.

Not sure what you mean there about the models.  What you describe is the 
opposite of my situation.  My controller does need the models.  That is 
exactly the problem, because I want to call that controller from another 
controller, but to do so I need the model files to be run.
 

> The other option is simply to avoid re-running the DB setup code and 
> instead use the already defined db object, passing it to the module code 
> from the other app:
>
> app1/models/db.py:
>
> db = DAL(...)
>
> app2/modules/db_definer.py:
>
> def define_tables(db):
> db.define_table('my_app2_table', ...)
>
> app1/controllers/default.py:
>
> from app2.modules.db_definer import define_tables
>
> def myfunc():
> define_tables(db)
> ...
>
> The above assumes both apps are using the same database, but if they are 
> not, then there is no "re-running the DB setup code" anyway, as you would 
> be dealing with two different databases, which would each need their own 
> setup code.
>  
>

What already-defined DB object?  If I use exec_environment, model files 
aren't run, so there is no db object.  In fact both controllers are using 
the same models, but if I use exec_environment, the object I get doesn't 
know about any models.  I could put code to create the models in the 
controller function (or in the controller top-level), which seems to be 
what your example is doing), but then I am, again, re-running that 
model-creation code twice (once in the original controller and once in the 
exec-ed controller).

Incidentally, I tried to just inject the db object into the object I get 
from exec_environment, but even this is difficult.  The reason is that 
exec_environment executes the file in a dict namespace, but then what it 
returns is not that dict, but a Storage object that is a (shallow) copy of 
it.  So if I do "c = exec_environment('some/controller.py')", the object c 
is not in fact the global namespace of the execed file, but a copy of it.  
Normally I would be able to do "c.db = db" to inject db as a global 
variable, but instead I had to do tricky things like 
"c.some_function.func_globals['db'] = db".

Anyway, your response makes me think maybe I haven't explained my situation 
correctly.  What I have this: controllerA.py and controllerB both needs the 
model in modelA.py  controllerB wants to call a function in controllerA.  
What I want is a way, from within controller B, to say "run controllerA, 
making sure all the models it needs are available to it, and then give me 
controllerA as a module object (or something equivalent), so that I can 
call controllerA.some_function() and get the results".  All without any 
additional network requests.  (In fact, some_function is a service, so I 
will pass arguments to it.). 

The problem is that I don't want these internal cross-app calls to be 
>> considered "requests"; I just want them to be calls to controller 
>> functions, so anything like requires_https should be ignored (or just 
>> proxied to the "real" request which is making this sub-call to another app).
>>
>
> It should only take effect for HTTP requests, not executions via the 
> shell, scheduler, or cron.
>  
>

As I mentioned, the problem is that exec_environment does not run 

[web2py] Re: Call functions from other applications without network requests

[Anthony]

>
> In your example above, it's one model defining tables from another model.  
> But in my case, what I have is a controller function that needs access to 
> the db, which is defined in the model file.  But I found the hard way (and 
> later also from an old thread: 
> https://groups.google.com/forum/#!topic/web2py-developers/5TPI9oYOZHM ) 
> that exec_environment does not load models.  So this puts me kind of back 
> to where I was.
>
> The simplest solution I found was to put, in the controller file, code 
> that looks like this:
>
> from applications.otherapp.modules.db_definer
> db = db_definer.define_db()
>
> Then every time my controller is run, it will redefine the DB, using the 
> code set up in the other app's module to do this in a consistent way (along 
> the lines you suggested).
>
> What I don't like about this is that it means the DB-setup code is run 
> multiple times on a "normal" request.  That is, if I have code to set up 
> the DB in my model file, and I visit my db-accessing controller, it runs 
> the code in the model file to set up the DB, then runs the code again when 
> it runs the controller file.  I imagine this is not a huge penalty but it 
> is annoying.
>

So, then don't do it that way. You have two options. First, if you have a 
controller that doesn't need any of the model definitions from your 
application's model files, then you should probably re-architect your 
application so only the models that are needed get defined when they are 
needed (either using the conditional models functionality or by defining 
all models in modules that are imported where needed).

The other option is simply to avoid re-running the DB setup code and 
instead use the already defined db object, passing it to the module code 
from the other app:

app1/models/db.py:

db = DAL(...)

app2/modules/db_definer.py:

def define_tables(db):
db.define_table('my_app2_table', ...)

app1/controllers/default.py:

from app2.modules.db_definer import define_tables

def myfunc():
define_tables(db)
...

The above assumes both apps are using the same database, but if they are 
not, then there is no "re-running the DB setup code" anyway, as you would 
be dealing with two different databases, which would each need their own 
setup code.
 

> I do have a couple questions about this: First, am I right that the 
> performance impact of re-running the DAL object creation is negligible?  Is 
> it something I should be concerned about?
>

The impact should be minimal, but see above for alternatives.
 

>   Second, is there anything wrong with using request.requires_https() in 
> the top-level code of a model file?
>

That's typically where it would be -- usually one of the first lines in the 
first model file to avoid unnecessary processing (as it results in a 
redirect). This is assuming you want the whole app over HTTPS.

The problem is that I don't want these internal cross-app calls to be 
> considered "requests"; I just want them to be calls to controller 
> functions, so anything like requires_https should be ignored (or just 
> proxied to the "real" request which is making this sub-call to another app).
>

It should only take effect for HTTP requests, not executions via the shell, 
scheduler, or cron.
 

> The upshot of all this appears to be that there is no real way to get 
> web2py to give me access to the model/controller combination without an 
> HTTP request.  In web2py the models and controllers are tightly coupled to 
> the HTTP request.  I find this somewhat irritating, as to my mind in an MVC 
> framework the models and controllers shouldn't be so closely tied to the 
> transport mechanism.  That is, I should be able to say "use this model and 
> this controller and give me the data", without having to involve HTTP at 
> all.  It seems that, in web2py this coupling is encouraged because 
> controller functions directly access objects like request and response to 
> get their arguments, rather than having a separate "routing" mechanism that 
> maps HTTP request parameters to Python functions.  (This is what the 
> "Service" function does, but it requires a level of indirection with a 
> "call" entry point.)
>
 
>
In essence, the way I think of it, when a request comes in, it is routed to 
> a controller function.  That controller then runs along with its associated 
> models. and the controller function is called.  It returns some data, which 
> is passed to a view.  What I would like is the ability to do just the 
> middle part of that: take a controller, load its models, and run a 
> function.  No HTTP.  No request.  No response.  No network.  No view.  No 
> nothing.  Just the controller and the model.  For now I have a way to do 
> it, but it involves duplicating the model-controller linkage that web2py 
> already does (by "manually" recreating the DAL object from within the 
> controller).
>

If you need to share code across applications, I suggest you put that code 
into modules and 

[web2py] Re: Call functions from other applications without network requests

[Brendan Barnwell]

On Friday, January 13, 2017 at 8:21:48 AM UTC-8, Anthony wrote:
>
>
> You can use the DAL from modules as well. As Niphlod suggested, you can 
>>> also use the scheduler to schedule and run a task in the context of another 
>>> app, though that might not be as fast as you'd like, as even setting 
>>> immediate=True, it could take up to "heartbeat" seconds for the worker to 
>>> pick up and execute the task, and then you have to check for the results.
>>>
>>>
>> I can use the DAL, but can I use the "db" object defined in my models 
>> file?  Or do you mean I could use the DAL if I rewrote my other app to use 
>> a "model-less" approach where the models were defined in some other manner?
>>
>
> Yes, the point was that you can put whatever you want into modules, which 
> means the code can be shared across apps. If you want to share the code to 
> create a database connection object, then put that code in a module. Here 
> is the basic idea:
>
> /app1/modules/db.py:
>
> import os
> from gluon import *
>
> def dal(**kwargs):
> web2py_folder = current.request.env.applications_parent
> db_folder = os.path.abspath(os.path.join(web2py_folder, 'applications'
> ,
>  'app1', 'databases'))
> return DAL('sqlite://storage.sqlite', folder=db_folder, **kwargs)
>
> def define_tables(db):
> db.define_table('table1', Field('field1'))
>
> /app2/models/db.py:
>
> from applications.app1.modules.db import dal, define_tables
>
> app1_db = dal()
> define_tables(app1_db)
>
> # Now use app1_db as usual.
>
> Of course, the module defining the database connection and tables doesn't 
> have to be in the /modules folder of any particular application, nor does 
> the folder specified as the "folder" argument -- that can all be 
> centralized somewhere if desired.
>

Okay.  So I've been experimenting with different approaches here.

In your example above, it's one model defining tables from another model.  
But in my case, what I have is a controller function that needs access to 
the db, which is defined in the model file.  But I found the hard way (and 
later also from an old thread: 
https://groups.google.com/forum/#!topic/web2py-developers/5TPI9oYOZHM ) 
that exec_environment does not load models.  So this puts me kind of back 
to where I was.

The simplest solution I found was to put, in the controller file, code that 
looks like this:

from applications.otherapp.modules.db_definer
db = db_definer.define_db()

Then every time my controller is run, it will redefine the DB, using the 
code set up in the other app's module to do this in a consistent way (along 
the lines you suggested).

What I don't like about this is that it means the DB-setup code is run 
multiple times on a "normal" request.  That is, if I have code to set up 
the DB in my model file, and I visit my db-accessing controller, it runs 
the code in the model file to set up the DB, then runs the code again when 
it runs the controller file.  I imagine this is not a huge penalty but it 
is annoying.

I tried the suggestion from that old thread to use gluon.shell.env, but 
that did not work for me.  gluon.shell.env seems to have two main 
problems.  First, it stomps on the "current" object, causing the 
newly-created environment to have a new request/response/etc. and losing 
the old ones from the original request.  Second, the new request apparently 
isn't considered as an HTTPS request, which caused problems for me because 
my models file has request.requires_https().

I do have a couple questions about this: First, am I right that the 
performance impact of re-running the DAL object creation is negligible?  Is 
it something I should be concerned about?  Second, is there anything wrong 
with using request.requires_https() in the top-level code of a model file?  
This part of the codebase was done by someone else, so if that is a misuse 
of web2py I could potentially try to get it changed.  I think the intent 
was to ensure that every single request must be HTTPS.  The problem is that 
I don't want these internal cross-app calls to be considered "requests"; I 
just want them to be calls to controller functions, so anything like 
requires_https should be ignored (or just proxied to the "real" request 
which is making this sub-call to another app).

The upshot of all this appears to be that there is no real way to get 
web2py to give me access to the model/controller combination without an 
HTTP request.  In web2py the models and controllers are tightly coupled to 
the HTTP request.  I find this somewhat irritating, as to my mind in an MVC 
framework the models and controllers shouldn't be so closely tied to the 
transport mechanism.  That is, I should be able to say "use this model and 
this controller and give me the data", without having to involve HTTP at 
all.  It seems that, in web2py this coupling is encouraged because 
controller functions directly access objects like request and response to 
get their 

[web2py] Re: Call functions from other applications without network requests

[Anthony]

> You can use the DAL from modules as well. As Niphlod suggested, you can 
>> also use the scheduler to schedule and run a task in the context of another 
>> app, though that might not be as fast as you'd like, as even setting 
>> immediate=True, it could take up to "heartbeat" seconds for the worker to 
>> pick up and execute the task, and then you have to check for the results.
>>
>>
> I can use the DAL, but can I use the "db" object defined in my models 
> file?  Or do you mean I could use the DAL if I rewrote my other app to use 
> a "model-less" approach where the models were defined in some other manner?
>

Yes, the point was that you can put whatever you want into modules, which 
means the code can be shared across apps. If you want to share the code to 
create a database connection object, then put that code in a module. Here 
is the basic idea:

/app1/modules/db.py:

import os
from gluon import *

def dal(**kwargs):
web2py_folder = current.request.env.applications_parent
db_folder = os.path.abspath(os.path.join(web2py_folder, 'applications',
 'app1', 'databases'))
return DAL('sqlite://storage.sqlite', folder=db_folder, **kwargs)

def define_tables(db):
db.define_table('table1', Field('field1'))

/app2/models/db.py:

from applications.app1.modules.db import dal, define_tables

app1_db = dal()
define_tables(app1_db)

# Now use app1_db as usual.

Of course, the module defining the database connection and tables doesn't 
have to be in the /modules folder of any particular application, nor does 
the folder specified as the "folder" argument -- that can all be 
centralized somewhere if desired.

Also, note there is a way to share models defined in the model files of 
another app, though with limitations. See 
http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#Using-DAL-without-define-tables.
 
That approach can be used to access model definitions from app1 model files 
from within app2 model files. However, the model definitions within app2 
will only include the table and field attributes that can be gleaned from 
the *.table migration files in app1's /databases folder, which only 
includes attributes needed for the database schema definition (i.e., it 
doesn't include any web2py-specific attributes, such as validators, labels, 
represent functions, etc.).

Anthony

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Brendan Barnwell]

On Saturday, January 7, 2017 at 11:07:36 AM UTC-8, Anthony wrote:
>
> On Saturday, January 7, 2017 at 1:57:36 PM UTC-5, Brendan Barnwell wrote:
>>
>> The shared functionality involves DB access, so as far as I can tell it 
>> cannot be abstracted into modules that would then be imported with a normal 
>> import.
>>
>
> You can use the DAL from modules as well. As Niphlod suggested, you can 
> also use the scheduler to schedule and run a task in the context of another 
> app, though that might not be as fast as you'd like, as even setting 
> immediate=True, it could take up to "heartbeat" seconds for the worker to 
> pick up and execute the task, and then you have to check for the results.
>
>
I can use the DAL, but can I use the "db" object defined in my models 
file?  Or do you mean I could use the DAL if I rewrote my other app to use 
a "model-less" approach where the models were defined in some other manner?

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Anthony]

On Saturday, January 7, 2017 at 1:57:36 PM UTC-5, Brendan Barnwell wrote:
>
> The shared functionality involves DB access, so as far as I can tell it 
> cannot be abstracted into modules that would then be imported with a normal 
> import.
>

You can use the DAL from modules as well. As Niphlod suggested, you can 
also use the scheduler to schedule and run a task in the context of another 
app, though that might not be as fast as you'd like, as even setting 
immediate=True, it could take up to "heartbeat" seconds for the worker to 
pick up and execute the task, and then you have to check for the results.

Anthony

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Brendan Barnwell]

The shared functionality involves DB access, so as far as I can tell it 
cannot be abstracted into modules that would then be imported with a normal 
import.  But I will take a look at the exec_environment thing, which may be 
what I'm after.

On Thursday, January 5, 2017 at 8:26:49 AM UTC-8, Anthony wrote:
>
> Check out 
> http://web2py.com/books/default/chapter/29/04/the-core#Execution-environment 
> and http://web2py.com/books/default/chapter/29/04/the-core#Cooperation.
>
> You might also consider whether the shared functionality can be abstracted 
> into Python modules that can be imported by multiple applications.
>
> Anthony
>
> On Thursday, January 5, 2017 at 12:54:59 AM UTC-5, Brendan Barnwell wrote:
>>
>> I have a situation where I have (or am going to have) multiple 
>> applications running on a single web2py instance.  Each application will do 
>> its own thing, but they are all using the same databases.  Some of the 
>> applications may serve as backend API endpoints where certain kinds of data 
>> can be retrieved.  Other applications may sometimes want to make calls to 
>> these endpoints to get the data and use it in their own operations.
>>
>> So what I'm wondering is: is there any way for one application to call a 
>> controller in another application and get the raw data that WOULD be used 
>> to build an actual HTTP response, but without actually going through the 
>> network?  That is, suppose I have and endpoint /app1/controller1/get_data 
>> and another /app2/controller2/do_something .  Inside do_something, I want 
>> to get the data that get_data gets.
>>
>> Currently I have get_data set up as a JSON service.  I could certainly 
>> make the request to the other endpoint, but that involves a number of extra 
>> steps: generating an outgoing HTTP request, serializing any data to passed 
>> with it, generating the incoming HTTP request object, deserializing the 
>> request data, then serializing the response and deserializing it again when 
>> it's received.  What I really want to do is to just call the function 
>> get_data AS IF a request had been submitted, but have everything happen on 
>> the server, without any network requests actually happening.  As far as I 
>> can tell, I can't in any way "import" the get_data function from the other 
>> app, because if I call it, it won't have the magic environment set up 
>> (e.g., "db" won't exist for it to do its database queries).  I would even 
>> be okay with "pretending" like I was doing a request, as long as web2py 
>> could somehow realize that the app I want to call is running on the same 
>> web2py instance, and call it "directly" instead of submitting a network 
>> request to itself.
>>
>> I realize that this architecture involves apps that are more 
>> interdependent than they would be in a "by the book" web2py project.  But 
>> basically the idea is that the functionality of the apps is independent, 
>> and in theory they should work if running on different servers, by making 
>> real requests.  It's just that, IF they are running on the same server, I'd 
>> like to communicate between them without going through the network, to 
>> improve performance.  Also, I can't really change the architecture since 
>> it's for a project I'm working on but don't ultimately control.
>>
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Anthony]

Check out 
http://web2py.com/books/default/chapter/29/04/the-core#Execution-environment 
and http://web2py.com/books/default/chapter/29/04/the-core#Cooperation.

You might also consider whether the shared functionality can be abstracted 
into Python modules that can be imported by multiple applications.

Anthony

On Thursday, January 5, 2017 at 12:54:59 AM UTC-5, Brendan Barnwell wrote:
>
> I have a situation where I have (or am going to have) multiple 
> applications running on a single web2py instance.  Each application will do 
> its own thing, but they are all using the same databases.  Some of the 
> applications may serve as backend API endpoints where certain kinds of data 
> can be retrieved.  Other applications may sometimes want to make calls to 
> these endpoints to get the data and use it in their own operations.
>
> So what I'm wondering is: is there any way for one application to call a 
> controller in another application and get the raw data that WOULD be used 
> to build an actual HTTP response, but without actually going through the 
> network?  That is, suppose I have and endpoint /app1/controller1/get_data 
> and another /app2/controller2/do_something .  Inside do_something, I want 
> to get the data that get_data gets.
>
> Currently I have get_data set up as a JSON service.  I could certainly 
> make the request to the other endpoint, but that involves a number of extra 
> steps: generating an outgoing HTTP request, serializing any data to passed 
> with it, generating the incoming HTTP request object, deserializing the 
> request data, then serializing the response and deserializing it again when 
> it's received.  What I really want to do is to just call the function 
> get_data AS IF a request had been submitted, but have everything happen on 
> the server, without any network requests actually happening.  As far as I 
> can tell, I can't in any way "import" the get_data function from the other 
> app, because if I call it, it won't have the magic environment set up 
> (e.g., "db" won't exist for it to do its database queries).  I would even 
> be okay with "pretending" like I was doing a request, as long as web2py 
> could somehow realize that the app I want to call is running on the same 
> web2py instance, and call it "directly" instead of submitting a network 
> request to itself.
>
> I realize that this architecture involves apps that are more 
> interdependent than they would be in a "by the book" web2py project.  But 
> basically the idea is that the functionality of the apps is independent, 
> and in theory they should work if running on different servers, by making 
> real requests.  It's just that, IF they are running on the same server, I'd 
> like to communicate between them without going through the network, to 
> improve performance.  Also, I can't really change the architecture since 
> it's for a project I'm working on but don't ultimately control.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Call functions from other applications without network requests

[Niphlod]

never though about just scheduling the functions via the scheduler ?

On Thursday, January 5, 2017 at 6:54:59 AM UTC+1, Brendan Barnwell wrote:
>
> I have a situation where I have (or am going to have) multiple 
> applications running on a single web2py instance.  Each application will do 
> its own thing, but they are all using the same databases.  Some of the 
> applications may serve as backend API endpoints where certain kinds of data 
> can be retrieved.  Other applications may sometimes want to make calls to 
> these endpoints to get the data and use it in their own operations.
>
> So what I'm wondering is: is there any way for one application to call a 
> controller in another application and get the raw data that WOULD be used 
> to build an actual HTTP response, but without actually going through the 
> network?  That is, suppose I have and endpoint /app1/controller1/get_data 
> and another /app2/controller2/do_something .  Inside do_something, I want 
> to get the data that get_data gets.
>
> Currently I have get_data set up as a JSON service.  I could certainly 
> make the request to the other endpoint, but that involves a number of extra 
> steps: generating an outgoing HTTP request, serializing any data to passed 
> with it, generating the incoming HTTP request object, deserializing the 
> request data, then serializing the response and deserializing it again when 
> it's received.  What I really want to do is to just call the function 
> get_data AS IF a request had been submitted, but have everything happen on 
> the server, without any network requests actually happening.  As far as I 
> can tell, I can't in any way "import" the get_data function from the other 
> app, because if I call it, it won't have the magic environment set up 
> (e.g., "db" won't exist for it to do its database queries).  I would even 
> be okay with "pretending" like I was doing a request, as long as web2py 
> could somehow realize that the app I want to call is running on the same 
> web2py instance, and call it "directly" instead of submitting a network 
> request to itself.
>
> I realize that this architecture involves apps that are more 
> interdependent than they would be in a "by the book" web2py project.  But 
> basically the idea is that the functionality of the apps is independent, 
> and in theory they should work if running on different servers, by making 
> real requests.  It's just that, IF they are running on the same server, I'd 
> like to communicate between them without going through the network, to 
> improve performance.  Also, I can't really change the architecture since 
> it's for a project I'm working on but don't ultimately control.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.