Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-17 Thread josip povreslo 
Yep, that's where I've stopped with my research for now. Mysqlclient 
handles relatively ok for now although I think as well it's a problem with 
network there, wouldn't be the first time. Eithter way, thanks for your 
support, I'll get back with full details if/when I completely understand 
what's happening and how to configure the whole thing. 

Best,
Josip

On Monday, July 17, 2017 at 3:30:16 PM UTC+2, Mike Bayer wrote:
>
> On Mon, Jul 17, 2017 at 4:23 AM, josip povreslo 
>  wrote: 
> > To add more information to it: 
> > 
> > We are running our dB on AWS RDS, this is what MySQL say in logs when we 
> > switched to mysql-connector: 
>
> Oh.  That is an *extremely* different situation.Amazon RDS is a 
> MySQL variant that runs in an entirely Amazon-specific environment. 
> They are likely performing some kind of simple rate limiting on your 
> service, or otherwise having their own kinds of outages.It is not 
> normal that they should have basic problems like this with off the 
> shelf MySQL client libraries, and issues like "can't connect" are not 
> normal in the MySQL world unless there are network issues.  I don't 
> have experience with RDS so you'd need to check with their support. 
>
>
>
> > 
> > [Note] Aborted connection 9191 to db: 'connstr' (Got an error reading 
> > communication packets) 
> > 
> > 
> > Best Regards, 
> > Josip 
> > 
> > 
> > On Monday, July 17, 2017 at 9:47:50 AM UTC+2, josip povreslo wrote: 
> >> 
> >> Mike, 
> >> 
> >> Just as an info, mysql-connector is a total NO for us, when I switched 
> to 
> >> mysql-connector we saw bigger problems and those happens much more 
> often. 
> >> PyMySQL is the last thing to switch to and try. For this processing 
> piece 
> >> I'm thinking to replace SQLAlchemy with a direct database driver usage, 
> do 
> >> you think that's the catch or this is a more general issue with the 
> drivers 
> >> itself? 
> >> 
> >> Best, 
> >> Josip 
> >> 
> >> On Friday, July 14, 2017 at 11:57:12 PM UTC+2, Mike Bayer wrote: 
> >>> 
> >>> that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and 
> >>> unmaintained) or mysqlclient (much better, if you have it installed 
> >>> and not conflicting w/ MySQL-Python).   You'd need to specify 
> >>> "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend 
> >>> trying "mysql+pymysql://" to see if you get better results. 
> >>> 
> >>> as far as "can't connect to server" you should make sure you are not 
> >>> surpassing your max_connections limit, though it should be emitting 
> >>> different error message for that. 
> >>> 
> >>> The Queue.Empty error is not the "error", it has to do with how the 
> >>> QueuePool works internally to test that the pool is empty, and Python 
> >>> 3 can't help but report every exception in the chain.   The error is 
> >>> the can't connect to server. 
> >>> 
> >>> 
> >>> 
> >>> On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo 
> >>>  wrote: 
> >>> > In addition to the previous email, what I do is the following: 
> >>> > 
> >>> > When request comes in: 
> >>> > self.mysql_engine = 
> >>> > 
> >>> > 
> create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'], 
>
> >>> > self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index], 
> >>> > self.db_params['dbname']), pool_recycle=3600, 
> >>> > connect_args={'connect_timeout': 2}) 
> >>> > 
> >>> > And then when need to interact with the dB: 
> >>> > 
> >>> > connection = self.mysql_engine.connect() 
> >>> > connection.execute("raw query") 
> >>> > connection.close() 
> >>> > 
> >>> > 
> >>> > Best Regards, 
> >>> > Josip 
> >>> > 
> >>> > On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote: 
> >>> >> 
> >>> >> Hello, 
> >>> >> 
> >>> >> As mentioned in the subject, we have the following packages 
> installed: 
> >>> >> 
> >>> >> Python 3.5 
> >>> >> mysql-connector==2.1.6 
> >>> >> SQLAlchemy==1.1.11 
> >>> >> 
> >>> >> Our service is working until we get a bit more traffic (40 - 50 
> >>> >> req/min), 
> >>> >> although I'm still not 100% if it's a pattern, however, then we get 
> >>> >> this 
> >>> >> error: 
> >>> >> 
> >>> >> Traceback (most recent call last): 
> >>> >>   File 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> >>> >> line 
> >>> >> 1122, in _do_get 
> >>> >> return self._pool.get(wait, self._timeout) 
> >>> >>   File 
> >>> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", 
> >>> >> line 
> >>> >> 145, in get 
> >>> >> raise Empty 
> >>> >> sqlalchemy.util.queue.Empty 
> >>> >> During handling of the above exception, another exception occurred: 
> >>> >> Traceback (most recent call last): 
> >>> >>   File 
> >>> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", 
> >>> >> line 
> >>> >> 2147, in _wrap_pool_connect 
> >>> >> return fn() 
> >>> >>   File 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py",

Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-17 Thread Mike Bayer 
On Mon, Jul 17, 2017 at 4:23 AM, josip povreslo
 wrote:
> To add more information to it:
>
> We are running our dB on AWS RDS, this is what MySQL say in logs when we
> switched to mysql-connector:

Oh.  That is an *extremely* different situation.Amazon RDS is a
MySQL variant that runs in an entirely Amazon-specific environment.
They are likely performing some kind of simple rate limiting on your
service, or otherwise having their own kinds of outages.It is not
normal that they should have basic problems like this with off the
shelf MySQL client libraries, and issues like "can't connect" are not
normal in the MySQL world unless there are network issues.  I don't
have experience with RDS so you'd need to check with their support.



>
> [Note] Aborted connection 9191 to db: 'connstr' (Got an error reading
> communication packets)
>
>
> Best Regards,
> Josip
>
>
> On Monday, July 17, 2017 at 9:47:50 AM UTC+2, josip povreslo wrote:
>>
>> Mike,
>>
>> Just as an info, mysql-connector is a total NO for us, when I switched to
>> mysql-connector we saw bigger problems and those happens much more often.
>> PyMySQL is the last thing to switch to and try. For this processing piece
>> I'm thinking to replace SQLAlchemy with a direct database driver usage, do
>> you think that's the catch or this is a more general issue with the drivers
>> itself?
>>
>> Best,
>> Josip
>>
>> On Friday, July 14, 2017 at 11:57:12 PM UTC+2, Mike Bayer wrote:
>>>
>>> that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and
>>> unmaintained) or mysqlclient (much better, if you have it installed
>>> and not conflicting w/ MySQL-Python).   You'd need to specify
>>> "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend
>>> trying "mysql+pymysql://" to see if you get better results.
>>>
>>> as far as "can't connect to server" you should make sure you are not
>>> surpassing your max_connections limit, though it should be emitting
>>> different error message for that.
>>>
>>> The Queue.Empty error is not the "error", it has to do with how the
>>> QueuePool works internally to test that the pool is empty, and Python
>>> 3 can't help but report every exception in the chain.   The error is
>>> the can't connect to server.
>>>
>>>
>>>
>>> On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo
>>>  wrote:
>>> > In addition to the previous email, what I do is the following:
>>> >
>>> > When request comes in:
>>> > self.mysql_engine =
>>> >
>>> > create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'],
>>> > self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index],
>>> > self.db_params['dbname']), pool_recycle=3600,
>>> > connect_args={'connect_timeout': 2})
>>> >
>>> > And then when need to interact with the dB:
>>> >
>>> > connection = self.mysql_engine.connect()
>>> > connection.execute("raw query")
>>> > connection.close()
>>> >
>>> >
>>> > Best Regards,
>>> > Josip
>>> >
>>> > On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> As mentioned in the subject, we have the following packages installed:
>>> >>
>>> >> Python 3.5
>>> >> mysql-connector==2.1.6
>>> >> SQLAlchemy==1.1.11
>>> >>
>>> >> Our service is working until we get a bit more traffic (40 - 50
>>> >> req/min),
>>> >> although I'm still not 100% if it's a pattern, however, then we get
>>> >> this
>>> >> error:
>>> >>
>>> >> Traceback (most recent call last):
>>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py",
>>> >> line
>>> >> 1122, in _do_get
>>> >> return self._pool.get(wait, self._timeout)
>>> >>   File
>>> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py",
>>> >> line
>>> >> 145, in get
>>> >> raise Empty
>>> >> sqlalchemy.util.queue.Empty
>>> >> During handling of the above exception, another exception occurred:
>>> >> Traceback (most recent call last):
>>> >>   File
>>> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py",
>>> >> line
>>> >> 2147, in _wrap_pool_connect
>>> >> return fn()
>>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py",
>>> >> line
>>> >> 328, in unique_connection
>>> >> return _ConnectionFairy._checkout(self)
>>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py",
>>> >> line
>>> >> 766, in _checkout
>>> >> fairy = _ConnectionRecord.checkout(pool)
>>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py",
>>> >> line
>>> >> 516, in checkout
>>> >> rec = pool._do_get()
>>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py",
>>> >> line
>>> >> 1138, in _do_get
>>> >> self._dec_overflow()
>>> >>   File
>>> >>
>>> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
>>> >> line 66, in __exit__
>>> >> compat.reraise(exc_type, exc_value, exc_tb)
>>> >>   File
>>> >>

Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-17 Thread josip povreslo 
To add more information to it:

We are running our dB on AWS RDS, this is what MySQL say in logs when we 
switched to mysql-connector:

[Note] Aborted connection 9191 to db: 'connstr' (Got an error reading 
communication packets)


Best Regards,
Josip

On Monday, July 17, 2017 at 9:47:50 AM UTC+2, josip povreslo wrote:
>
> Mike,
>
> Just as an info, mysql-connector is a total NO for us, when I switched to 
> mysql-connector we saw bigger problems and those happens much more often. 
> PyMySQL is the last thing to switch to and try. For this processing piece 
> I'm thinking to replace SQLAlchemy with a direct database driver usage, do 
> you think that's the catch or this is a more general issue with the drivers 
> itself?
>
> Best,
> Josip
>
> On Friday, July 14, 2017 at 11:57:12 PM UTC+2, Mike Bayer wrote:
>>
>> that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and 
>> unmaintained) or mysqlclient (much better, if you have it installed 
>> and not conflicting w/ MySQL-Python).   You'd need to specify 
>> "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend 
>> trying "mysql+pymysql://" to see if you get better results. 
>>
>> as far as "can't connect to server" you should make sure you are not 
>> surpassing your max_connections limit, though it should be emitting 
>> different error message for that. 
>>
>> The Queue.Empty error is not the "error", it has to do with how the 
>> QueuePool works internally to test that the pool is empty, and Python 
>> 3 can't help but report every exception in the chain.   The error is 
>> the can't connect to server. 
>>
>>
>>
>> On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo 
>>  wrote: 
>> > In addition to the previous email, what I do is the following: 
>> > 
>> > When request comes in: 
>> > self.mysql_engine = 
>> > 
>> create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'], 
>>
>> > self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index], 
>> > self.db_params['dbname']), pool_recycle=3600, 
>> > connect_args={'connect_timeout': 2}) 
>> > 
>> > And then when need to interact with the dB: 
>> > 
>> > connection = self.mysql_engine.connect() 
>> > connection.execute("raw query") 
>> > connection.close() 
>> > 
>> > 
>> > Best Regards, 
>> > Josip 
>> > 
>> > On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote: 
>> >> 
>> >> Hello, 
>> >> 
>> >> As mentioned in the subject, we have the following packages installed: 
>> >> 
>> >> Python 3.5 
>> >> mysql-connector==2.1.6 
>> >> SQLAlchemy==1.1.11 
>> >> 
>> >> Our service is working until we get a bit more traffic (40 - 50 
>> req/min), 
>> >> although I'm still not 100% if it's a pattern, however, then we get 
>> this 
>> >> error: 
>> >> 
>> >> Traceback (most recent call last): 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 1122, in _do_get 
>> >> return self._pool.get(wait, self._timeout) 
>> >>   File 
>> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", 
>> line 
>> >> 145, in get 
>> >> raise Empty 
>> >> sqlalchemy.util.queue.Empty 
>> >> During handling of the above exception, another exception occurred: 
>> >> Traceback (most recent call last): 
>> >>   File 
>> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", 
>> line 
>> >> 2147, in _wrap_pool_connect 
>> >> return fn() 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 328, in unique_connection 
>> >> return _ConnectionFairy._checkout(self) 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 766, in _checkout 
>> >> fairy = _ConnectionRecord.checkout(pool) 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 516, in checkout 
>> >> rec = pool._do_get() 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 1138, in _do_get 
>> >> self._dec_overflow() 
>> >>   File 
>> >> 
>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py", 
>> >> line 66, in __exit__ 
>> >> compat.reraise(exc_type, exc_value, exc_tb) 
>> >>   File 
>> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", 
>> line 
>> >> 187, in reraise 
>> >> raise value 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 1135, in _do_get 
>> >> return self._create_connection() 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 333, in _create_connection 
>> >> return _ConnectionRecord(self) 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 461, in __init__ 
>> >> self.__connect(first_connect_check=True) 
>> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
>> line 
>> >> 651, in __connect 
>> >> connection =

Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-17 Thread josip povreslo 
Mike,

Just as an info, mysql-connector is a total NO for us, when I switched to 
mysql-connector we saw bigger problems and those happens much more often. 
PyMySQL is the last thing to switch to and try. For this processing piece 
I'm thinking to replace SQLAlchemy with a direct database driver usage, do 
you think that's the catch or this is a more general issue with the drivers 
itself?

Best,
Josip

On Friday, July 14, 2017 at 11:57:12 PM UTC+2, Mike Bayer wrote:
>
> that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and 
> unmaintained) or mysqlclient (much better, if you have it installed 
> and not conflicting w/ MySQL-Python).   You'd need to specify 
> "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend 
> trying "mysql+pymysql://" to see if you get better results. 
>
> as far as "can't connect to server" you should make sure you are not 
> surpassing your max_connections limit, though it should be emitting 
> different error message for that. 
>
> The Queue.Empty error is not the "error", it has to do with how the 
> QueuePool works internally to test that the pool is empty, and Python 
> 3 can't help but report every exception in the chain.   The error is 
> the can't connect to server. 
>
>
>
> On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo 
>  wrote: 
> > In addition to the previous email, what I do is the following: 
> > 
> > When request comes in: 
> > self.mysql_engine = 
> > 
> create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'], 
>
> > self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index], 
> > self.db_params['dbname']), pool_recycle=3600, 
> > connect_args={'connect_timeout': 2}) 
> > 
> > And then when need to interact with the dB: 
> > 
> > connection = self.mysql_engine.connect() 
> > connection.execute("raw query") 
> > connection.close() 
> > 
> > 
> > Best Regards, 
> > Josip 
> > 
> > On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote: 
> >> 
> >> Hello, 
> >> 
> >> As mentioned in the subject, we have the following packages installed: 
> >> 
> >> Python 3.5 
> >> mysql-connector==2.1.6 
> >> SQLAlchemy==1.1.11 
> >> 
> >> Our service is working until we get a bit more traffic (40 - 50 
> req/min), 
> >> although I'm still not 100% if it's a pattern, however, then we get 
> this 
> >> error: 
> >> 
> >> Traceback (most recent call last): 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 1122, in _do_get 
> >> return self._pool.get(wait, self._timeout) 
> >>   File 
> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", 
> line 
> >> 145, in get 
> >> raise Empty 
> >> sqlalchemy.util.queue.Empty 
> >> During handling of the above exception, another exception occurred: 
> >> Traceback (most recent call last): 
> >>   File 
> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", 
> line 
> >> 2147, in _wrap_pool_connect 
> >> return fn() 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 328, in unique_connection 
> >> return _ConnectionFairy._checkout(self) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 766, in _checkout 
> >> fairy = _ConnectionRecord.checkout(pool) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 516, in checkout 
> >> rec = pool._do_get() 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 1138, in _do_get 
> >> self._dec_overflow() 
> >>   File 
> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py", 
> >> line 66, in __exit__ 
> >> compat.reraise(exc_type, exc_value, exc_tb) 
> >>   File 
> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", 
> line 
> >> 187, in reraise 
> >> raise value 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 1135, in _do_get 
> >> return self._create_connection() 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 333, in _create_connection 
> >> return _ConnectionRecord(self) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 461, in __init__ 
> >> self.__connect(first_connect_check=True) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 651, in __connect 
> >> connection = pool._invoke_creator(self) 
> >>   File 
> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/strategies.py", 
> >> line 105, in connect 
> >> return dialect.connect(*cargs, **cparams) 
> >>   File 
> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/default.py", 
> >> line 393, in connect 
> >> return self.dbapi.connect(*cargs, **cparams) 
> >>   File "/usr/local/lib64/python3.5/site-packages/MySQLdb/__init__.py", 
> >> line

Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-14 Thread josip povreslo 
Thanks!

On Saturday, July 15, 2017 at 12:08:19 AM UTC+2, Mike Bayer wrote:
>
> I've set up 
> https://bitbucket.org/zzzeek/sqlalchemy/issues/4028/move-cant-connect-handling-outside-of
>  
> to deal with the misleading "pool.Empty" error for 1.2, though that is 
> not the problem you're having here. 
>
> On Fri, Jul 14, 2017 at 5:57 PM, Mike Bayer  > wrote: 
> > that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and 
> > unmaintained) or mysqlclient (much better, if you have it installed 
> > and not conflicting w/ MySQL-Python).   You'd need to specify 
> > "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend 
> > trying "mysql+pymysql://" to see if you get better results. 
> > 
> > as far as "can't connect to server" you should make sure you are not 
> > surpassing your max_connections limit, though it should be emitting 
> > different error message for that. 
> > 
> > The Queue.Empty error is not the "error", it has to do with how the 
> > QueuePool works internally to test that the pool is empty, and Python 
> > 3 can't help but report every exception in the chain.   The error is 
> > the can't connect to server. 
> > 
> > 
> > 
> > On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo 
> >  wrote: 
> >> In addition to the previous email, what I do is the following: 
> >> 
> >> When request comes in: 
> >> self.mysql_engine = 
> >> 
> create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'], 
>
> >> self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index], 
> >> self.db_params['dbname']), pool_recycle=3600, 
> >> connect_args={'connect_timeout': 2}) 
> >> 
> >> And then when need to interact with the dB: 
> >> 
> >> connection = self.mysql_engine.connect() 
> >> connection.execute("raw query") 
> >> connection.close() 
> >> 
> >> 
> >> Best Regards, 
> >> Josip 
> >> 
> >> On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote: 
> >>> 
> >>> Hello, 
> >>> 
> >>> As mentioned in the subject, we have the following packages installed: 
> >>> 
> >>> Python 3.5 
> >>> mysql-connector==2.1.6 
> >>> SQLAlchemy==1.1.11 
> >>> 
> >>> Our service is working until we get a bit more traffic (40 - 50 
> req/min), 
> >>> although I'm still not 100% if it's a pattern, however, then we get 
> this 
> >>> error: 
> >>> 
> >>> Traceback (most recent call last): 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 1122, in _do_get 
> >>> return self._pool.get(wait, self._timeout) 
> >>>   File 
> >>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", 
> line 
> >>> 145, in get 
> >>> raise Empty 
> >>> sqlalchemy.util.queue.Empty 
> >>> During handling of the above exception, another exception occurred: 
> >>> Traceback (most recent call last): 
> >>>   File 
> >>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", 
> line 
> >>> 2147, in _wrap_pool_connect 
> >>> return fn() 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 328, in unique_connection 
> >>> return _ConnectionFairy._checkout(self) 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 766, in _checkout 
> >>> fairy = _ConnectionRecord.checkout(pool) 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 516, in checkout 
> >>> rec = pool._do_get() 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 1138, in _do_get 
> >>> self._dec_overflow() 
> >>>   File 
> >>> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py", 
> >>> line 66, in __exit__ 
> >>> compat.reraise(exc_type, exc_value, exc_tb) 
> >>>   File 
> >>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", 
> line 
> >>> 187, in reraise 
> >>> raise value 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 1135, in _do_get 
> >>> return self._create_connection() 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 333, in _create_connection 
> >>> return _ConnectionRecord(self) 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 461, in __init__ 
> >>> self.__connect(first_connect_check=True) 
> >>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >>> 651, in __connect 
> >>> connection = pool._invoke_creator(self) 
> >>>   File 
> >>> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/strategies.py", 
> >>> line 105, in connect 
> >>> return dialect.connect(*cargs, **cparams) 
> >>>   File 
> >>> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/default.py", 
> >>> line 393, in connect 
> >>> return self.dbapi.connect(*cargs, **cparams) 
> >>>   File

Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-14 Thread josip povreslo 
Thanks Mike!

Point valid, it's mysqlclient==1.3.9, we don't have MySQL-Python installed, 
inherited the system so I assumed it's mysql-connector from some reason, 
nevertheless. 

Hmmm, I was afraid that's the actual case (connection error). We were 
checking the connection limits and that's not the case, will try to switch 
everything over to mysql-connector then and get back with the results. Will 
do it on Monday (not pushing things live over the weekend :)). 

Best Regards,
Josip


On Friday, July 14, 2017 at 11:57:12 PM UTC+2, Mike Bayer wrote:
>
> that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and 
> unmaintained) or mysqlclient (much better, if you have it installed 
> and not conflicting w/ MySQL-Python).   You'd need to specify 
> "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend 
> trying "mysql+pymysql://" to see if you get better results. 
>
> as far as "can't connect to server" you should make sure you are not 
> surpassing your max_connections limit, though it should be emitting 
> different error message for that. 
>
> The Queue.Empty error is not the "error", it has to do with how the 
> QueuePool works internally to test that the pool is empty, and Python 
> 3 can't help but report every exception in the chain.   The error is 
> the can't connect to server. 
>
>
>
> On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo 
>  wrote: 
> > In addition to the previous email, what I do is the following: 
> > 
> > When request comes in: 
> > self.mysql_engine = 
> > 
> create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'], 
>
> > self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index], 
> > self.db_params['dbname']), pool_recycle=3600, 
> > connect_args={'connect_timeout': 2}) 
> > 
> > And then when need to interact with the dB: 
> > 
> > connection = self.mysql_engine.connect() 
> > connection.execute("raw query") 
> > connection.close() 
> > 
> > 
> > Best Regards, 
> > Josip 
> > 
> > On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote: 
> >> 
> >> Hello, 
> >> 
> >> As mentioned in the subject, we have the following packages installed: 
> >> 
> >> Python 3.5 
> >> mysql-connector==2.1.6 
> >> SQLAlchemy==1.1.11 
> >> 
> >> Our service is working until we get a bit more traffic (40 - 50 
> req/min), 
> >> although I'm still not 100% if it's a pattern, however, then we get 
> this 
> >> error: 
> >> 
> >> Traceback (most recent call last): 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 1122, in _do_get 
> >> return self._pool.get(wait, self._timeout) 
> >>   File 
> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", 
> line 
> >> 145, in get 
> >> raise Empty 
> >> sqlalchemy.util.queue.Empty 
> >> During handling of the above exception, another exception occurred: 
> >> Traceback (most recent call last): 
> >>   File 
> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", 
> line 
> >> 2147, in _wrap_pool_connect 
> >> return fn() 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 328, in unique_connection 
> >> return _ConnectionFairy._checkout(self) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 766, in _checkout 
> >> fairy = _ConnectionRecord.checkout(pool) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 516, in checkout 
> >> rec = pool._do_get() 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 1138, in _do_get 
> >> self._dec_overflow() 
> >>   File 
> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py", 
> >> line 66, in __exit__ 
> >> compat.reraise(exc_type, exc_value, exc_tb) 
> >>   File 
> >> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", 
> line 
> >> 187, in reraise 
> >> raise value 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 1135, in _do_get 
> >> return self._create_connection() 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 333, in _create_connection 
> >> return _ConnectionRecord(self) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 461, in __init__ 
> >> self.__connect(first_connect_check=True) 
> >>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", 
> line 
> >> 651, in __connect 
> >> connection = pool._invoke_creator(self) 
> >>   File 
> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/strategies.py", 
> >> line 105, in connect 
> >> return dialect.connect(*cargs, **cparams) 
> >>   File 
> >> 
> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/default.py", 
> >> line 393, in connect 
> >> return self.dbapi.connect(*cargs, **cparams) 
>

Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-14 Thread Mike Bayer 
I've set up 
https://bitbucket.org/zzzeek/sqlalchemy/issues/4028/move-cant-connect-handling-outside-of
to deal with the misleading "pool.Empty" error for 1.2, though that is
not the problem you're having here.

On Fri, Jul 14, 2017 at 5:57 PM, Mike Bayer  wrote:
> that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and
> unmaintained) or mysqlclient (much better, if you have it installed
> and not conflicting w/ MySQL-Python).   You'd need to specify
> "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend
> trying "mysql+pymysql://" to see if you get better results.
>
> as far as "can't connect to server" you should make sure you are not
> surpassing your max_connections limit, though it should be emitting
> different error message for that.
>
> The Queue.Empty error is not the "error", it has to do with how the
> QueuePool works internally to test that the pool is empty, and Python
> 3 can't help but report every exception in the chain.   The error is
> the can't connect to server.
>
>
>
> On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo
>  wrote:
>> In addition to the previous email, what I do is the following:
>>
>> When request comes in:
>> self.mysql_engine =
>> create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'],
>> self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index],
>> self.db_params['dbname']), pool_recycle=3600,
>> connect_args={'connect_timeout': 2})
>>
>> And then when need to interact with the dB:
>>
>> connection = self.mysql_engine.connect()
>> connection.execute("raw query")
>> connection.close()
>>
>>
>> Best Regards,
>> Josip
>>
>> On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote:
>>>
>>> Hello,
>>>
>>> As mentioned in the subject, we have the following packages installed:
>>>
>>> Python 3.5
>>> mysql-connector==2.1.6
>>> SQLAlchemy==1.1.11
>>>
>>> Our service is working until we get a bit more traffic (40 - 50 req/min),
>>> although I'm still not 100% if it's a pattern, however, then we get this
>>> error:
>>>
>>> Traceback (most recent call last):
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 1122, in _do_get
>>> return self._pool.get(wait, self._timeout)
>>>   File
>>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", line
>>> 145, in get
>>> raise Empty
>>> sqlalchemy.util.queue.Empty
>>> During handling of the above exception, another exception occurred:
>>> Traceback (most recent call last):
>>>   File
>>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", line
>>> 2147, in _wrap_pool_connect
>>> return fn()
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 328, in unique_connection
>>> return _ConnectionFairy._checkout(self)
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 766, in _checkout
>>> fairy = _ConnectionRecord.checkout(pool)
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 516, in checkout
>>> rec = pool._do_get()
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 1138, in _do_get
>>> self._dec_overflow()
>>>   File
>>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
>>> line 66, in __exit__
>>> compat.reraise(exc_type, exc_value, exc_tb)
>>>   File
>>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", line
>>> 187, in reraise
>>> raise value
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 1135, in _do_get
>>> return self._create_connection()
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 333, in _create_connection
>>> return _ConnectionRecord(self)
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 461, in __init__
>>> self.__connect(first_connect_check=True)
>>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>>> 651, in __connect
>>> connection = pool._invoke_creator(self)
>>>   File
>>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/strategies.py",
>>> line 105, in connect
>>> return dialect.connect(*cargs, **cparams)
>>>   File
>>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/default.py",
>>> line 393, in connect
>>> return self.dbapi.connect(*cargs, **cparams)
>>>   File "/usr/local/lib64/python3.5/site-packages/MySQLdb/__init__.py",
>>> line 81, in Connect
>>> return Connection(*args, **kwargs)
>>>   File "/usr/local/lib64/python3.5/site-packages/MySQLdb/connections.py",
>>> line 191, in __init__
>>> super(Connection, self).__init__(*args, **kwargs2)
>>> _mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server
>>> on 'dbhost_obviously' (4)")
>>>
>>> We haven't been seeing this error up until recently when we ported our
>>> service to

Re: [sqlalchemy] Re: Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

2017-07-14 Thread Mike Bayer 
that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and
unmaintained) or mysqlclient (much better, if you have it installed
and not conflicting w/ MySQL-Python).   You'd need to specify
"mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend
trying "mysql+pymysql://" to see if you get better results.

as far as "can't connect to server" you should make sure you are not
surpassing your max_connections limit, though it should be emitting
different error message for that.

The Queue.Empty error is not the "error", it has to do with how the
QueuePool works internally to test that the pool is empty, and Python
3 can't help but report every exception in the chain.   The error is
the can't connect to server.



On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo
 wrote:
> In addition to the previous email, what I do is the following:
>
> When request comes in:
> self.mysql_engine =
> create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'],
> self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index],
> self.db_params['dbname']), pool_recycle=3600,
> connect_args={'connect_timeout': 2})
>
> And then when need to interact with the dB:
>
> connection = self.mysql_engine.connect()
> connection.execute("raw query")
> connection.close()
>
>
> Best Regards,
> Josip
>
> On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote:
>>
>> Hello,
>>
>> As mentioned in the subject, we have the following packages installed:
>>
>> Python 3.5
>> mysql-connector==2.1.6
>> SQLAlchemy==1.1.11
>>
>> Our service is working until we get a bit more traffic (40 - 50 req/min),
>> although I'm still not 100% if it's a pattern, however, then we get this
>> error:
>>
>> Traceback (most recent call last):
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 1122, in _do_get
>> return self._pool.get(wait, self._timeout)
>>   File
>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", line
>> 145, in get
>> raise Empty
>> sqlalchemy.util.queue.Empty
>> During handling of the above exception, another exception occurred:
>> Traceback (most recent call last):
>>   File
>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", line
>> 2147, in _wrap_pool_connect
>> return fn()
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 328, in unique_connection
>> return _ConnectionFairy._checkout(self)
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 766, in _checkout
>> fairy = _ConnectionRecord.checkout(pool)
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 516, in checkout
>> rec = pool._do_get()
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 1138, in _do_get
>> self._dec_overflow()
>>   File
>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
>> line 66, in __exit__
>> compat.reraise(exc_type, exc_value, exc_tb)
>>   File
>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", line
>> 187, in reraise
>> raise value
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 1135, in _do_get
>> return self._create_connection()
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 333, in _create_connection
>> return _ConnectionRecord(self)
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 461, in __init__
>> self.__connect(first_connect_check=True)
>>   File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line
>> 651, in __connect
>> connection = pool._invoke_creator(self)
>>   File
>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/strategies.py",
>> line 105, in connect
>> return dialect.connect(*cargs, **cparams)
>>   File
>> "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/default.py",
>> line 393, in connect
>> return self.dbapi.connect(*cargs, **cparams)
>>   File "/usr/local/lib64/python3.5/site-packages/MySQLdb/__init__.py",
>> line 81, in Connect
>> return Connection(*args, **kwargs)
>>   File "/usr/local/lib64/python3.5/site-packages/MySQLdb/connections.py",
>> line 191, in __init__
>> super(Connection, self).__init__(*args, **kwargs2)
>> _mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server
>> on 'dbhost_obviously' (4)")
>>
>> We haven't been seeing this error up until recently when we ported our
>> service to Python 3.5. I've read most of the articles related to this queue
>> empty error and most of those are from '12 or '13 and mentioning immature
>> mysql drivers. Is this still the case, should we search for an alternative
>> to mysql-connector or alternative to something else? Any help or guidance is
>> appreciated!
>>
>> Thanks!
>>
>> Best,
>> Josip
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper