Yep, that's where I've stopped with my research for now. Mysqlclient handles relatively ok for now although I think as well it's a problem with network there, wouldn't be the first time. Eithter way, thanks for your support, I'll get back with full details if/when I completely understand what's happening and how to configure the whole thing.
Best, Josip On Monday, July 17, 2017 at 3:30:16 PM UTC+2, Mike Bayer wrote: > > On Mon, Jul 17, 2017 at 4:23 AM, josip povreslo > <josip.p...@gmail.com <javascript:>> wrote: > > To add more information to it: > > > > We are running our dB on AWS RDS, this is what MySQL say in logs when we > > switched to mysql-connector: > > Oh. That is an *extremely* different situation. Amazon RDS is a > MySQL variant that runs in an entirely Amazon-specific environment. > They are likely performing some kind of simple rate limiting on your > service, or otherwise having their own kinds of outages. It is not > normal that they should have basic problems like this with off the > shelf MySQL client libraries, and issues like "can't connect" are not > normal in the MySQL world unless there are network issues. I don't > have experience with RDS so you'd need to check with their support. > > > > > > > [Note] Aborted connection 9191 to db: 'connstr' (Got an error reading > > communication packets) > > > > > > Best Regards, > > Josip > > > > > > On Monday, July 17, 2017 at 9:47:50 AM UTC+2, josip povreslo wrote: > >> > >> Mike, > >> > >> Just as an info, mysql-connector is a total NO for us, when I switched > to > >> mysql-connector we saw bigger problems and those happens much more > often. > >> PyMySQL is the last thing to switch to and try. For this processing > piece > >> I'm thinking to replace SQLAlchemy with a direct database driver usage, > do > >> you think that's the catch or this is a more general issue with the > drivers > >> itself? > >> > >> Best, > >> Josip > >> > >> On Friday, July 14, 2017 at 11:57:12 PM UTC+2, Mike Bayer wrote: > >>> > >>> that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and > >>> unmaintained) or mysqlclient (much better, if you have it installed > >>> and not conflicting w/ MySQL-Python). You'd need to specify > >>> "mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend > >>> trying "mysql+pymysql://" to see if you get better results. > >>> > >>> as far as "can't connect to server" you should make sure you are not > >>> surpassing your max_connections limit, though it should be emitting > >>> different error message for that. > >>> > >>> The Queue.Empty error is not the "error", it has to do with how the > >>> QueuePool works internally to test that the pool is empty, and Python > >>> 3 can't help but report every exception in the chain. The error is > >>> the can't connect to server. > >>> > >>> > >>> > >>> On Fri, Jul 14, 2017 at 5:41 PM, josip povreslo > >>> <josip.p...@gmail.com> wrote: > >>> > In addition to the previous email, what I do is the following: > >>> > > >>> > When request comes in: > >>> > self.mysql_engine = > >>> > > >>> > > create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'], > > >>> > self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index], > >>> > self.db_params['dbname']), pool_recycle=3600, > >>> > connect_args={'connect_timeout': 2}) > >>> > > >>> > And then when need to interact with the dB: > >>> > > >>> > connection = self.mysql_engine.connect() > >>> > connection.execute("raw query") > >>> > connection.close() > >>> > > >>> > > >>> > Best Regards, > >>> > Josip > >>> > > >>> > On Friday, July 14, 2017 at 11:36:26 PM UTC+2, josip povreslo wrote: > >>> >> > >>> >> Hello, > >>> >> > >>> >> As mentioned in the subject, we have the following packages > installed: > >>> >> > >>> >> Python 3.5 > >>> >> mysql-connector==2.1.6 > >>> >> SQLAlchemy==1.1.11 > >>> >> > >>> >> Our service is working until we get a bit more traffic (40 - 50 > >>> >> req/min), > >>> >> although I'm still not 100% if it's a pattern, however, then we get > >>> >> this > >>> >> error: > >>> >> > >>> >> Traceback (most recent call last): > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 1122, in _do_get > >>> >> return self._pool.get(wait, self._timeout) > >>> >> File > >>> >> > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", > >>> >> line > >>> >> 145, in get > >>> >> raise Empty > >>> >> sqlalchemy.util.queue.Empty > >>> >> During handling of the above exception, another exception occurred: > >>> >> Traceback (most recent call last): > >>> >> File > >>> >> > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", > >>> >> line > >>> >> 2147, in _wrap_pool_connect > >>> >> return fn() > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 328, in unique_connection > >>> >> return _ConnectionFairy._checkout(self) > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 766, in _checkout > >>> >> fairy = _ConnectionRecord.checkout(pool) > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 516, in checkout > >>> >> rec = pool._do_get() > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 1138, in _do_get > >>> >> self._dec_overflow() > >>> >> File > >>> >> > >>> >> > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py", > >>> >> line 66, in __exit__ > >>> >> compat.reraise(exc_type, exc_value, exc_tb) > >>> >> File > >>> >> > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", > >>> >> line > >>> >> 187, in reraise > >>> >> raise value > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 1135, in _do_get > >>> >> return self._create_connection() > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 333, in _create_connection > >>> >> return _ConnectionRecord(self) > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 461, in __init__ > >>> >> self.__connect(first_connect_check=True) > >>> >> File > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", > >>> >> line > >>> >> 651, in __connect > >>> >> connection = pool._invoke_creator(self) > >>> >> File > >>> >> > >>> >> > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/strategies.py", > >>> >> line 105, in connect > >>> >> return dialect.connect(*cargs, **cparams) > >>> >> File > >>> >> > >>> >> > "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/default.py", > >>> >> line 393, in connect > >>> >> return self.dbapi.connect(*cargs, **cparams) > >>> >> File > "/usr/local/lib64/python3.5/site-packages/MySQLdb/__init__.py", > >>> >> line 81, in Connect > >>> >> return Connection(*args, **kwargs) > >>> >> File > >>> >> "/usr/local/lib64/python3.5/site-packages/MySQLdb/connections.py", > >>> >> line 191, in __init__ > >>> >> super(Connection, self).__init__(*args, **kwargs2) > >>> >> _mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL > >>> >> server > >>> >> on 'dbhost_obviously' (4)") > >>> >> > >>> >> We haven't been seeing this error up until recently when we ported > our > >>> >> service to Python 3.5. I've read most of the articles related to > this > >>> >> queue > >>> >> empty error and most of those are from '12 or '13 and mentioning > >>> >> immature > >>> >> mysql drivers. Is this still the case, should we search for an > >>> >> alternative > >>> >> to mysql-connector or alternative to something else? Any help or > >>> >> guidance is > >>> >> appreciated! > >>> >> > >>> >> Thanks! > >>> >> > >>> >> Best, > >>> >> Josip > >>> > > >>> > -- > >>> > SQLAlchemy - > >>> > The Python SQL Toolkit and Object Relational Mapper > >>> > > >>> > http://www.sqlalchemy.org/ > >>> > > >>> > To post example code, please provide an MCVE: Minimal, Complete, and > >>> > Verifiable Example. See http://stackoverflow.com/help/mcve for a > full > >>> > description. > >>> > --- > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "sqlalchemy" group. > >>> > To unsubscribe from this group and stop receiving emails from it, > send > >>> > an > >>> > email to sqlalchemy+...@googlegroups.com. > >>> > To post to this group, send email to sqlal...@googlegroups.com. > >>> > Visit this group at https://groups.google.com/group/sqlalchemy. > >>> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > SQLAlchemy - > > The Python SQL Toolkit and Object Relational Mapper > > > > http://www.sqlalchemy.org/ > > > > To post example code, please provide an MCVE: Minimal, Complete, and > > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > > description. > > --- > > You received this message because you are subscribed to the Google > Groups > > "sqlalchemy" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to sqlalchemy+...@googlegroups.com <javascript:>. > > To post to this group, send email to sqlal...@googlegroups.com > <javascript:>. > > Visit this group at https://groups.google.com/group/sqlalchemy. > > For more options, visit https://groups.google.com/d/optout. > -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.