Re: [sqlalchemy] pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread John Doe
On Wednesday, April 22, 2015 at 3:20:13 PM UTC+1, Michael Bayer wrote: a COMMIT every time would slow it down, yes, but then in your profiling you'd see the do_commit() method taking up that time as well. if you can get your logging going, you'll see what SQL its emitting, and you can

Re: [sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Richard Gerd Kuesters
well, i know that repeating (query) is somehow strange, i believe the main reason is that *compile()**.process*are not bound to the query anymore (*i might be wrong*). well, for me it was a one time solution, perhaps a little digging can bring you a better approach :) cheers, richard. On

Re: [sqlalchemy] pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread Mike Bayer
On 4/22/15 5:45 AM, John Doe wrote: Is there a way to output all the SQL statements into a file? I don't mean a logging file with other information, I mean a file with **only** SQL statements, which could ideally be run as it is from within my database client. Python logging will get you

Re: [sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Richard Gerd Kuesters
yeah, well, that's much simpler. in my scenario, where i had to use this piece of code, literal_binds are necessary since the query is somehow huge, with lots of alias and parameters :) On 04/22/2015 11:07 AM, Mike Bayer wrote: On 4/22/15 8:27 AM, Richard Gerd Kuesters wrote: hi, you

Re: [sqlalchemy] SAWarning: Unicode type received non-unicode bind param value

2015-04-22 Thread Mike Bayer
On 4/22/15 5:47 AM, Pavel S wrote: If I turn warnings into errors, the problem is still the same: 1) where the error occured as the code path in traceback is not pointing to the place where select() has been bound with values that would suggest your code has some less typical form where you

Re: [sqlalchemy] pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread Mike Bayer
On 4/22/15 10:11 AM, John Doe wrote: On Wednesday, April 22, 2015 at 2:46:19 PM UTC+1, Michael Bayer wrote: I carefully re-read your first email, and now I see, that you are trying to write *TO* the database, yes?And you're comparing the speed of writing *TO* to the

Re: [sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Pavel S
Hi Richard, thanks, your solution works. ( I don't need formatted output) However using *query* twice in the expression looks to me a bit awkward. Isn't there some shortcut? Dne středa 22. dubna 2015 14:27:41 UTC+2 Richard Kuesters napsal(a): hi, you must use a specific dialect so

Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.

2015-04-22 Thread Mike Bayer
On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var

Re: [sqlalchemy] pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread John Doe
On Wednesday, April 22, 2015 at 2:46:19 PM UTC+1, Michael Bayer wrote: I carefully re-read your first email, and now I see, that you are trying to write *TO* the database, yes?And you're comparing the speed of writing *TO* to the speed of reading *FROM*? Unfortunately, selecting

Re: [sqlalchemy] SAWarning: Unicode type received non-unicode bind param value

2015-04-22 Thread Mike Bayer
On 4/22/15 3:48 AM, Pavel S wrote: Hi, it happened to me many times during development, mainly when used custom column types, that I passed wrong type of value to the query. Then the the following warning was emitted:| || SAWarning: Unicodetype received non-unicode bindparam value | The

Re: [sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Pavel S
Hi Mike, your solutions works too. Thanks to you both! Dne středa 22. dubna 2015 16:07:28 UTC+2 Michael Bayer napsal(a): On 4/22/15 8:27 AM, Richard Gerd Kuesters wrote: hi, you must use a specific dialect so sqlalchemy can create it for you. not the best usage, imho, but here it

Re: [sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Mike Bayer
On 4/22/15 8:27 AM, Richard Gerd Kuesters wrote: hi, you must use a specific dialect so sqlalchemy can create it for you. not the best usage, imho, but here it goes: *stmt = query.compile().process(query, literal_binds=True)* OK since Pavel noted the double compile, if you want the

Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.

2015-04-22 Thread Mike Bayer
On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var

Re: [sqlalchemy] Can I restrict the set of mappers that configure_mappers() works on?

2015-04-22 Thread Steven Winfield
Sort of... it goes something like this: In some database (call it code) there is a table - script(name text, contents text) - which is wrapped using an sqlalchemy class called Script. In our environment we fire up fresh python interpreters, import the library that defines Script, install a

Re: [sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Jonathan Vanasco
Just be warned of this method -- make sure you're using the most recent SqlAlchemy version in the .9x or 1.x branches. Earlier versions would not apply LIMIT or OFFSET into the bind. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe

Re: [sqlalchemy] pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread Mike Bayer
On 4/22/15 12:07 PM, John Doe wrote: On Wednesday, April 22, 2015 at 3:20:13 PM UTC+1, Michael Bayer wrote: a COMMIT every time would slow it down, yes, but then in your profiling you'd see the do_commit() method taking up that time as well. if you can get your logging

Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.

2015-04-22 Thread Mike Bayer
On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var

[sqlalchemy] Re: pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread Jonathan Vanasco
I've ran into similar issues like this before -- but on different databases. I think it's bad to think of the problem as transmitting 31k/s -- which assumes a particular issue; and much better to think of it as processing 31k/s, which gives a lot more room for interpretation. Looking on

Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.

2015-04-22 Thread Guido Winkelmann
On Wednesday 22 April 2015 14:17:02 Mike Bayer wrote: On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var

[sqlalchemy] Re: pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread John Doe
On Wednesday, April 22, 2015 at 7:19:08 PM UTC+1, Jonathan Vanasco wrote: [...] I'm guessing that this issue is with the driver. Here's a semi-related thread: * http://stackoverflow.com/questions/5693885/pyodbc-very-slow-bulk-insert-speed

[sqlalchemy] SAWarning: Unicode type received non-unicode bind param value

2015-04-22 Thread Pavel S
Hi, it happened to me many times during development, mainly when used custom column types, that I passed wrong type of value to the query. Then the the following warning was emitted: SAWarning: Unicode type received non-unicode bindparam value The problem with such warning is it does not say

Re: [sqlalchemy] SAWarning: Unicode type received non-unicode bind param value

2015-04-22 Thread Sebastian Elsner | RISE
Hey, you dan do: import warnings warnings.simplefilter('|error|') This will raise an exception. and give you a stacktrace on where the Unicode warnign happened. On 04/22/2015 09:48 AM, Pavel S wrote: Hi, it happened to me many times during development, mainly when used custom column

[sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Pavel S
Hello, I have pythonic application which imports custom module written in C++ using boost::python. The module creates database connection(s) and executes queries. The python calls various methods on that module and passes plain SQL into them. Results are then returned to python. I would like

Re: [sqlalchemy] pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread John Doe
Is there a way to output all the SQL statements into a file? I don't mean a logging file with other information, I mean a file with **only** SQL statements, which could ideally be run as it is from within my database client. I tried setting echo=True, and I also tried this:

Re: [sqlalchemy] SAWarning: Unicode type received non-unicode bind param value

2015-04-22 Thread Pavel S
If I turn warnings into errors, the problem is still the same: 1) where the error occured as the code path in traceback is not pointing to the place where select() has been bound with values 2) you don't know what was the value Dne středa 22. dubna 2015 10:09:51 UTC+2 SElsner napsal(a):

Re: [sqlalchemy] SQLAlchemy as plain-SQL generator

2015-04-22 Thread Richard Gerd Kuesters
hi, you must use a specific dialect so sqlalchemy can create it for you. not the best usage, imho, but here it goes: *stmt = query.compile().process(query, literal_binds=True)* i don't know if you want it formated or what, if so, *sqlparse*provides a good way to do it. but, again, this