Thanks for your interest Dave.
Yes the files are big, the last one was 8mb with about 44k records
About the scheduler. i start it manually at the control panel launching the
script below ("python scheduler.py" from a pythonanywhere console). After a
few days it becomes unresponsive and i have to kill the console and restart
it. Also pythonanywhere use to stops their servers 30 minutes once in a
month for mantainance.
Im using a template they provided. (guess thats to avoid multiple instances)
#/usr/bin/env python
import os
import socket
import subprocess
import sys
filename = os.path.abspath(__file__) # we use this to generate a unique
socket name
# we use a local socket as a lock.
# it can only be bound once, and will be released if the process dies
# we want to keep the socket open until the very end of
# our script so we use a global variable to avoid going
# out of scope and being garbage-collected
lock_socket = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
try:
lock_socket.bind('\0' + filename)
except socket.error:
print("Failed to acquire lock, task must already be running")
sys.exit()
subprocess.call(["python", "web2py/web2py.py", "-K", "myapp"])
I also have the problem of concurrency, each user should be able to import
their owns csv files. This cannot be achieved by the scheduler in a sane
way. You have to declare the how many workers you need in advance (i dont
know how many users are gonna upload files at the same time) and if i
declare like lets say, 3 or 4 workers, im adding a lot of overhead to the
db because of the heartbeats and task's outputs.
I have just did the same test with those 40k records on a custom thread as
the first post, not using dal, but mysql.connector so the thread has its
own connection, it took 10 seconds, when the scheduler takes 30+minutes.
The question remains, is there anyway to use dal instead mysql.connector?
Ive try this inside the thread's __init__
self.db = DAL(cnnstring, ..., migrate=False)
self.db.define_table(fields.., migrate=False)
but doesnt work, i get the error above.
Thanks
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
