[sqlalchemy] test email

[Mike Bayer]

this is a test.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/287c068c-1194-41a9-82b3-b57fdde3c702%40app.fastmail.com.

Re: [sqlalchemy] Composite columns and None

[Mike Bayer]

On Tue, Nov 14, 2023, at 6:02 PM, jens.t...@gmail.com wrote:
> Hello everyone,
> 
> I wanted to follow up on the examples on Composite Column Types 
>  and in particular 
> setting the mapped composite value in Python-land to *None*.
> 
> For instance, this class
> *@dataclasses*.dataclass
> *class* *Point*:
> x: int
> y: int
> is used in the following composite mapping and I’d like the two mapped 
> properties to be optional (for the sake of the argument, whether it makes 
> sense or not):
> *class* *Vertex*(Base):
> start: Mapped[Point] | None = composite(mapped_column("x1"), 
> mapped_column("y1"))
> end: Mapped[Point] | None = composite(mapped_column("x2"), 
> mapped_column("y2"))The optional would then have to map to *nullable=True* 
> for both mapped columns, which in turn would need to be checked to ensure 
> integrity — either both or neither of the two mapped columns for a *Point* 
> can be NULL at the same time.
> 
> I wasn’t able to find a recipe 
>  for this use case. 
> What’s the recommendation to implement this? Will I have to roll most of this 
> manually, or does SQLA provide support?

you can create this using a custom callable for the actual composite type, 
where you would be bypassing the new-style annotations and dataclass detection 
part of things and relying on the older style of declaration.   You'd add a 
composite_values to your dataclass as well and just use the old style

import dataclasses

from sqlalchemy import create_engine
from sqlalchemy import Integer
from sqlalchemy import select
from sqlalchemy.orm import composite
from sqlalchemy.orm import DeclarativeBase
from sqlalchemy.orm import Mapped
from sqlalchemy.orm import mapped_column
from sqlalchemy.orm import Session


@dataclasses.dataclass
class Point:
x: int | None
y: int | None

def __composite_values__(self):
return (self.x, self.y)


class Base(DeclarativeBase):
pass


def nullable_point(x, y):
if x is None and y is None:
return None
else:
return Point(x, y)


class Vertex(Base):
__tablename__ = "vertices"

id: Mapped[int] = mapped_column(primary_key=True)

start: Mapped[Point | None] = composite(
nullable_point,
mapped_column("x1", Integer, nullable=True),
mapped_column("y1", Integer, nullable=True),
)
end: Mapped[Point | None] = composite(
nullable_point,
mapped_column("x2", Integer, nullable=True),
mapped_column("y2", Integer, nullable=True),
)

def __repr__(self):
return f"Vertex(start={self.start}, end={self.end})"


e = create_engine("sqlite://", echo=True)
Base.metadata.create_all(e)

s = Session(e)

s.add_all(
[
Vertex(start=Point(5, 10)),
Vertex(start=Point(10, None), end=Point(25, 17)),
]
)
s.commit()

for v in s.scalars(select(Vertex)):
print(f"{v.start} {v.end}")





> 
> Much thanks!
> Jens
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/e0a3f019-11d0-4156-9c13-0a0e493dc40an%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/8da01a71-032d-4c95-82fb-b27970be111e%40app.fastmail.com.

Re: [sqlalchemy] sqlalchemy 2.0 and ABCMeta

[Mike Bayer]

hi -

I'm not sure what the issue is but if you are tinkering with metaclasses, we 
first off have an alternate version of DeclarativeBase called 
DeclarativeBaseNoMeta that has no metaclass installed, and may be a better 
place to build off custom metaclass solutions as you wont need to be 
subclassing the internal metaclass used by DeclarativeBase. 

I would advise reading the source code for the DeclarativeBase / 
DeclarativeBaseNoMeta classes, paying special attention to the 
__init_subclass__() method which is where Declarative now does its work, and 
ensuring any __init_subclass__ method on your own class is working as expected 
and not interfering with the one called by SQLAlchemy's class (or just vendor 
the method yourself if necessary).



On Thu, Nov 9, 2023, at 3:04 AM, 'Iwan Vosloo' via sqlalchemy wrote:
> Hi there,
>
> We are migrating our code from SqlAlchemy 1.4 to 2.0 (2.0.23 to be 
> specific).
>
> We have had the following, which allowed some classes inheriting from 
> our Base to use an ABCMeta metaclass:
>
> ---
> class DeclarativeABCMeta(DeclarativeMeta, ABCMeta):
>  pass
>
> metadata = MetaData(naming_convention=naming_convention)
> Base = declarative_base(metadata=metadata, metaclass=DeclarativeABCMeta)
> ---
>
> The code above works fine on 2.0, but if we want heed the 2.0 docs that 
> comment that declarative_base is superceded by using a class inheriting 
> from DeclarativeBase, we ought to have something like:
>
> ---
> class DeclarativeABCMeta(DeclarativeMeta, ABCMeta):
>  pass
>
> metadata = MetaData(naming_convention=naming_convention)
>
> class Base(DeclarativeBase, metaclass=DeclarativeABCMeta):
>  """A Base for using with declarative."""
>  __abstract__ = True
>  metadata = metadata
> ---
>
> This, however breaks when it hits the first class inheriting from Base:
>
> ---
> class SchemaVersion(Base):
>  __tablename__ = 'reahl_schema_version'
>  id = Column(Integer, primary_key=True)
>  version = Column(String(50))
>  egg_name = Column(String(80))
> ---
>
> With:
>
> [site-packages]/sqlalchemy/orm/decl_api.py:195: in __init__
>  _as_declarative(reg, cls, dict_)
> [site-packages]/sqlalchemy/orm/decl_base.py:247: in _as_declarative
>  return _MapperConfig.setup_mapping(registry, cls, dict_, None, {})
> [site-packages]/sqlalchemy/orm/decl_base.py:328: in setup_mapping
>  return _ClassScanMapperConfig(
> [site-packages]/sqlalchemy/orm/decl_base.py:520: in __init__
>  super().__init__(registry, cls_, mapper_kw)
> [site-packages]/sqlalchemy/orm/decl_base.py:344: in __init__
>  instrumentation.register_class(
> [site-packages]/sqlalchemy/orm/instrumentation.py:684: in register_class
>  manager._update_state(
> [site-packages]/sqlalchemy/orm/instrumentation.py:209: in _update_state
>  registry._add_manager(self)
> [site-packages]/sqlalchemy/orm/decl_api.py:1380: in _add_manager
>  raise exc.ArgumentError(
> E   sqlalchemy.exc.ArgumentError: Class ' 'reahl.sqlalchemysupport.sqlalchemysupport.SchemaVersion'>' already has 
> a primary mapper defined.
>
> Any ideas on what we are doing wrong here?
>
> Thanks
> Iwan
>
> -- 
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/5d616bf0-ffa7-4061-adaf-cf1c7577e0fc%40reahl.org.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/8ae83685-03c3-4b80-b2a3-c4d228d94626%40app.fastmail.com.

Re: [sqlalchemy] Issue with type signature of DBAPI Connection

[Mike Bayer]

there does seem to be an extra Sequence on the outside that should be removed, 
please open an issue.


On Tue, Nov 7, 2023, at 9:46 PM, William Hakim wrote:
> I was recently using the DBAPI Cursor, and it seems to me that the type 
> signature of the `executemany()` function is incorrect:
> 
> https://github.com/sqlalchemy/sqlalchemy/blob/main/lib/sqlalchemy/engine/interfaces.py#L194
> 
> The given type signature for the second argument to `executemany()` is ` 
> "Sequence[Sequence[Sequence[Any]] | Sequence[Mapping[str, Any]]]"`, however 
> this doesn't appear right (and doesn't actually work) - it seems like it 
> should instead be `Sequence[Sequence[Any]] | Sequence[Mapping[str, Any]]`.
> 
> Happy to open a Pull Request if this is indeed an issue.
> 
> Will
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/048482ea-0c27-46e3-9acf-6257d685f855n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/4898aaa6-94da-4bfc-baab-d537688ced68%40app.fastmail.com.

Re: [sqlalchemy] Connection issue with URL-encoded passwords in `do-connect` event

[Mike Bayer]

On Fri, Nov 3, 2023, at 7:41 PM, SeJun Bae wrote:
> Hello everyone,
> I have encountered an odd behavior when using URL-encoded tokens as passwords 
> for connections with Postgres; my application connects to a Postgres AWS RDS 
> instance using a token that expires (IAM Authentication 
> ).
>  Specifically, if I set `cparams['password']` to a URL-encoded value in the 
> `do_connect` event, the connection fails. However, it succeeds if the value 
> isn't URL encoded. This seems inconsistent with how the engine handles 
> URL-encoded values. Is this the intended behavior ?

the password needs to be url encoded, *when embedded into the URL*. When you 
handle the password as a single value in the parameters sent to the DBAPI, this 
needs to be the actual password, and not URL encoded, since it's not being 
extracted from a URL. This is consistent with the API for the URL object 
itself, where you have the option to create a URL progammatically from 
non-encoded fields (see 
https://docs.sqlalchemy.org/en/20/core/engines.html#creating-urls-programmatically).
  hope this helps



> If not, would the team be open to a PR that calls `unquote` on the 
> `cparams['password'] when it's changed to ensure consistency ? 
>  
> Here is a minimal example to illustrate the issue:
> 
> from sqlalchemy.orm import sessionmaker
> from sqlalchemy import create_engine, event, text
> from urllib.parse import quote
> 
> *# Assume there is a db user `test` with some initial token
> POSTGRES_HOST = 'localhost'
> PORT = '5432'
> USERNAME = 'test'
> NEW_PASSWORD = '%2FH'
> 
> engine1 = 
> create_engine(f"postgresql://{USERNAME}:some_token_that_expires@{POSTGRES_HOST}:{PORT}/esrf",
>  echo=True)
> 
> @event.listens_for(engine1, 'do_connect')
> def receive_do_connect(dialect, conn_rec, cargs, cparams):
> # This doesn't work.
> cparams['password'] = quote(NEW_PASSWORD)
> 
> with engine1.connect() as connection:
> connection.execute(text('SELECT 1;'))
> 
> engine2 = 
> create_engine(f"postgresql://{USERNAME}:some_token_that_expires@{POSTGRES_HOST}:{PORT}/esrf",
>  echo=True)
> 
> @event.listens_for(engine2, 'do_connect')
> def receive_do_connect(dialect, conn_rec, cargs, cparams):
> # This works
> cparams['password'] = NEW_PASSWORD
> 
> with engine2.connect() as connection:
> connection.execute(text('SELECT 1;'))*
> 
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/6a124e8b-fabc-47b6-8e68-db939acc98f7n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/49e6a38c-c329-4d77-ab6c-a12fd9076a69%40app.fastmail.com.

Re: [sqlalchemy] Postgresq Execute Many with Textual SQL Convenience Issue

[Mike Bayer]

On Thu, Nov 2, 2023, at 11:24 AM, mkmo...@gmail.com wrote:
> Hi Mike,
> 
> When using Core, we can do a bulk insert and bulk return with Postgresql 
> trivially:
> 
> from sqlalchemy import table, column
> t = table('foo', column('id'), column('bar')
> 
> ins = t.insert().values([{'bar': 'a'}, {'bar': 'b'}]).returning(foo.id)
> 
> results = conn.execute(ins)
>
> ids = results.fetchall()
> 
> However, with raw textual SQL, it is a bit more inconvenient.
> 
> The following doesn't work:
> 
> results = conn.execute(text(
> 'insert into foo values (:bar) returning id
> ), [{'bar': 'a'}, {'bar': 'b'}])
>
> # raises sqlalchemy.exc.ResourceClosedError: This result object does not 
> return rows. It has been closed automatically.
> results.fetchall()

That's because you're using excecutemany.   pep-249 executemany does not 
support rows being returned, so if you send a list of params like that, it's an 
automatic result close.  there's nothing to fetch.

I would also note that the statement you have above might not be as useful as 
you think since I would assume you would want those "RETURNING" rows to line up 
with the dictionaries you are sending.   This is not a guarantee for most 
backends and the SQL Server backend will actually actively rearrange the rows.  
 Special (extremely inconvenient) forms must be taken to ensure this ordering.

SQLAlchemy 2.0 supports a new form of INSERT called insertmanyvalues 
(https://docs.sqlalchemy.org/en/20/core/connections.html#insert-many-values-behavior-for-insert-statements
 ) which can receive a list of parameter dictionaries along with an insert() 
construct and convert the operation into series of batched single statements 
that are yielded as a single result set.   That is, it does **not** use DBAPI 
executemany (except on Oracle which has special support), it uses DBAPI 
execute.   It also does this very special and inconvenient formatting of the 
INSERT statement to ensure to the greatest degree possible that RETURNING rows 
are ordered the same way as the parameter sets.


> 
> To get it working, we have to do it this way:
> 
> results = conn.execute(text(
> 'insert into foo values (:bar0), (:bar1)
> ), {'bar0': 'x', 'bar1': 'y'})
> 
> assert results.fetchall()
> 
> 
> This isn't convenient. For example you would have to convert a list of bars 
> like [{'bar': 'a'}, {'bar': 'b'}] into a single dict with uniquely name keys 
> {'bar0': 'a', 'bar1': 'b'}.

you do, because there's no result set implied from an executemany, you have to 
use an execute.   

if you want SQLAlchemy to convert your multiple parameters into a series of 
"INSERT..VALUES", do the execution of them, and then with RETURNING work the 
results back together, that's exactly what insertmanyvalues does.We spent 
many months developing this feature plus figuring out the RETURNING / ordering 
thing which took an extra month, so you'd be tapping into a great deal of 
development efforts by using that feature.

However, that feature works only with insert() constructs. With text(), we 
have no idea what your statement says and we don't parse SQL.  you'd be on your 
own there.

> 
> I imagine sqlalchemy is doing that under the hood when using core. Is there 
> some convenience function available in sqlalchemy core that I can use to 
> simplify this?

The functions which do this start at 
https://github.com/sqlalchemy/sqlalchemy/blob/b51cccec6a953555f39c16005cb5a2a49a6f4b21/lib/sqlalchemy/engine/default.py#L758
  and then digs into compiler at 
https://github.com/sqlalchemy/sqlalchemy/blob/b51cccec6a953555f39c16005cb5a2a49a6f4b21/lib/sqlalchemy/sql/compiler.py#L5306
 , where you'll note these functions are now huge and complicated, as they must 
accommodate all cases amongst a wide variety of statements scenarios, typing 
issues on both the input and output side, and backends.

You can definitely write simple helpers to convert simple INSERT statements on 
your own here, or maybe look at psycopg2's fast execution helpers also at 
https://www.psycopg.org/docs/extras.html#fast-execution-helpers which are older 
/ simpler versions of this kind of thing (however still probably not 
generalizable).


-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/e408bc7b-efcd-407b-88f0-216fc9de1fc9%40app.fastmail.com.

Re: [sqlalchemy] Event listener for when query results get matched with pre-existing objects on the session

[Mike Bayer]

great, glad to know that new API is finding itself useful.

On Sun, Oct 22, 2023, at 9:32 PM, 'Tony Cosentini' via sqlalchemy wrote:
> do_orm_execute() (and freezing the results) totally allows for what I'm 
> trying to do! Also I forgot to mention earlier, but this is only going to run 
> in tests.
> 
> Thanks again for all the help,
> Tony
> 
> On Friday, October 20, 2023 at 11:10:23 PM UTC+8 Mike Bayer wrote:
>> 
>> 
>> On Fri, Oct 20, 2023, at 10:46 AM, 'Tony Cosentini' via sqlalchemy wrote:
>>> Oh I see, thanks for clarifying.
>>> 
>>> I'm trying to detect cases where we depend on the autoflush behavior. For 
>>> example, in the sample above, when the query runs with no_autoflush, we 
>>> won't get back any results (when previously we would get back one row). 
>>> It's a fairly large codebase, so I was trying to automate finding these 
>>> cases in order to add explicit flush calls when needed or just pass in the 
>>> pending object instead of running a query.
>> 
>> 
>> there's not really an event for an object that just gets returned by a query 
>> from the identity map.you would need to do something more drastic like a 
>> do_orm_execute() hook that runs queries internally and then looks at all the 
>> returned objects.  i wouldnt put that in production.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> 
>>> 
>>> Tony
>>> 
>>> On Fri, Oct 20, 2023 at 10:42 PM Mike Bayer  
>>> wrote:
>>>> __
>>>> 
>>>> 
>>>> On Fri, Oct 20, 2023, at 9:50 AM, 'Tony Cosentini' via sqlalchemy wrote:
>>>>> Weird, I did try that but I can't seem to trigger it.
>>>>> 
>>>>> Here's a self-contained test: 
>>>>> https://gist.github.com/tonycosentini/4dee3478695d032ca67707b5e26739b6
>>>> 
>>>> 
>>>> the object was not affected in that query (that is, not mutated).  if you 
>>>> change the query to this:
>>>> 
>>>>  table_one_instances = 
>>>> session.query(TableOne).populate_existing().all()
>>>> 
>>>> then the refresh event is triggered.
>>>> 
>>>> Im not really following what kind of bug you are trying to detect.
>>>> 
>>>> 
>>>>> 
>>>>> I have the event listener set to just crash, but it never triggers.
>>>>> 
>>>>> On Fri, Oct 20, 2023 at 9:20 PM Mike Bayer 
>>>>>  wrote:
>>>>>> __
>>>>>> 
>>>>>> 
>>>>>> On Fri, Oct 20, 2023, at 8:08 AM, 'Tony Cosentini' via sqlalchemy wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> Is there any way to listen for an event for when a query result gets 
>>>>>>> merged into a pre-existing object in the session?
>>>>>> 
>>>>>> this is the refresh event:
>>>>>> 
>>>>>> https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> I'm working on disabling autoflush for some of our codebase (mostly to 
>>>>>>> cut down on queries on a high QPS path), but before doing that I want 
>>>>>>> to write a utility to detect when we read data that was flushed via 
>>>>>>> autoflush to cut down on any potential bugs.
>>>>>>> 
>>>>>>> What I'd like to do is this (while autoflush is still enabled):
>>>>>>> 
>>>>>>> table_one = TableOne(name='test')
>>>>>>> session.add(table_one) # Track this object as pending a flush via the 
>>>>>>> before_* listeners, this is working as expected.
>>>>>>> 
>>>>>>> table_two_instances = session.query(TableTwo).all() # All good, doesn't 
>>>>>>> do anything with the table_one instance created earlier
>>>>>>> 
>>>>>>> table_one_instances = session.query(TableOne).all() # I would like to 
>>>>>>> log a warning here as the results of this query depend on a flush 
>>>>>>> happening. What I'm hoping to do is detect that one of the rows coming 
>>>>>>> back is the same object that was flagged earlier, but I can't see to 
>>>>>>> find the correct event to use.
>>>>>> 
>>

Re: [sqlalchemy] Event listener for when query results get matched with pre-existing objects on the session

[Mike Bayer]

On Fri, Oct 20, 2023, at 10:46 AM, 'Tony Cosentini' via sqlalchemy wrote:
> Oh I see, thanks for clarifying.
> 
> I'm trying to detect cases where we depend on the autoflush behavior. For 
> example, in the sample above, when the query runs with no_autoflush, we won't 
> get back any results (when previously we would get back one row). It's a 
> fairly large codebase, so I was trying to automate finding these cases in 
> order to add explicit flush calls when needed or just pass in the pending 
> object instead of running a query.


there's not really an event for an object that just gets returned by a query 
from the identity map.you would need to do something more drastic like a 
do_orm_execute() hook that runs queries internally and then looks at all the 
returned objects.  i wouldnt put that in production.







> 
> Tony
> 
> On Fri, Oct 20, 2023 at 10:42 PM Mike Bayer 
>  wrote:
>> __
>> 
>> 
>> On Fri, Oct 20, 2023, at 9:50 AM, 'Tony Cosentini' via sqlalchemy wrote:
>>> Weird, I did try that but I can't seem to trigger it.
>>> 
>>> Here's a self-contained test: 
>>> https://gist.github.com/tonycosentini/4dee3478695d032ca67707b5e26739b6
>> 
>> 
>> the object was not affected in that query (that is, not mutated).  if you 
>> change the query to this:
>> 
>>  table_one_instances = session.query(TableOne).populate_existing().all()
>> 
>> then the refresh event is triggered.
>> 
>> Im not really following what kind of bug you are trying to detect.
>> 
>> 
>>> 
>>> I have the event listener set to just crash, but it never triggers.
>>> 
>>> On Fri, Oct 20, 2023 at 9:20 PM Mike Bayer 
>>>  wrote:
>>>> __
>>>> 
>>>> 
>>>> On Fri, Oct 20, 2023, at 8:08 AM, 'Tony Cosentini' via sqlalchemy wrote:
>>>>> Hi,
>>>>> 
>>>>> Is there any way to listen for an event for when a query result gets 
>>>>> merged into a pre-existing object in the session?
>>>> 
>>>> this is the refresh event:
>>>> 
>>>> https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh
>>>> 
>>>> 
>>>>> 
>>>>> I'm working on disabling autoflush for some of our codebase (mostly to 
>>>>> cut down on queries on a high QPS path), but before doing that I want to 
>>>>> write a utility to detect when we read data that was flushed via 
>>>>> autoflush to cut down on any potential bugs.
>>>>> 
>>>>> What I'd like to do is this (while autoflush is still enabled):
>>>>> 
>>>>> table_one = TableOne(name='test')
>>>>> session.add(table_one) # Track this object as pending a flush via the 
>>>>> before_* listeners, this is working as expected.
>>>>> 
>>>>> table_two_instances = session.query(TableTwo).all() # All good, doesn't 
>>>>> do anything with the table_one instance created earlier
>>>>> 
>>>>> table_one_instances = session.query(TableOne).all() # I would like to log 
>>>>> a warning here as the results of this query depend on a flush happening. 
>>>>> What I'm hoping to do is detect that one of the rows coming back is the 
>>>>> same object that was flagged earlier, but I can't see to find the correct 
>>>>> event to use.
>>>> 
>>>> you probably want to also use refresh_flush also, which will be invoked 
>>>> for column defaults that are populated on the object
>>>> 
>>>> https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh_flush
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> Is this possible? I can't seem to find the appropriate event that would 
>>>>> trigger when the results from the .all() query get merged back in with 
>>>>> the existing objects in the session.
>>>> 
>>>> that's the refresh event
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> SQLAlchemy -
>>>> The Python SQL Toolkit and Object Relational Mapper
>>>>  
>>>> http://www.sqlalchemy.org/
>>>>  
>>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>>> description.
>>>> ---
>>

Re: [sqlalchemy] Event listener for when query results get matched with pre-existing objects on the session

[Mike Bayer]

On Fri, Oct 20, 2023, at 9:50 AM, 'Tony Cosentini' via sqlalchemy wrote:
> Weird, I did try that but I can't seem to trigger it.
> 
> Here's a self-contained test: 
> https://gist.github.com/tonycosentini/4dee3478695d032ca67707b5e26739b6


the object was not affected in that query (that is, not mutated).  if you 
change the query to this:

 table_one_instances = session.query(TableOne).populate_existing().all()

then the refresh event is triggered.

Im not really following what kind of bug you are trying to detect.


> 
> I have the event listener set to just crash, but it never triggers.
> 
> On Fri, Oct 20, 2023 at 9:20 PM Mike Bayer 
>  wrote:
>> __
>> 
>> 
>> On Fri, Oct 20, 2023, at 8:08 AM, 'Tony Cosentini' via sqlalchemy wrote:
>>> Hi,
>>> 
>>> Is there any way to listen for an event for when a query result gets merged 
>>> into a pre-existing object in the session?
>> 
>> this is the refresh event:
>> 
>> https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh
>> 
>> 
>>> 
>>> I'm working on disabling autoflush for some of our codebase (mostly to cut 
>>> down on queries on a high QPS path), but before doing that I want to write 
>>> a utility to detect when we read data that was flushed via autoflush to cut 
>>> down on any potential bugs.
>>> 
>>> What I'd like to do is this (while autoflush is still enabled):
>>> 
>>> table_one = TableOne(name='test')
>>> session.add(table_one) # Track this object as pending a flush via the 
>>> before_* listeners, this is working as expected.
>>> 
>>> table_two_instances = session.query(TableTwo).all() # All good, doesn't do 
>>> anything with the table_one instance created earlier
>>> 
>>> table_one_instances = session.query(TableOne).all() # I would like to log a 
>>> warning here as the results of this query depend on a flush happening. What 
>>> I'm hoping to do is detect that one of the rows coming back is the same 
>>> object that was flagged earlier, but I can't see to find the correct event 
>>> to use.
>> 
>> you probably want to also use refresh_flush also, which will be invoked for 
>> column defaults that are populated on the object
>> 
>> https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh_flush
>> 
>> 
>> 
>> 
>>> 
>>> Is this possible? I can't seem to find the appropriate event that would 
>>> trigger when the results from the .all() query get merged back in with the 
>>> existing objects in the session.
>> 
>> that's the refresh event
>> 
>> 
>> 
>> 
>> -- 
>> SQLAlchemy - 
>> The Python SQL Toolkit and Object Relational Mapper
>>  
>> http://www.sqlalchemy.org/
>>  
>> To post example code, please provide an MCVE: Minimal, Complete, and 
>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>> description.
>> --- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "sqlalchemy" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/sqlalchemy/9bevy3Kpql4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> sqlalchemy+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/sqlalchemy/c5b89f67-e3fa-45f2-8774-463d154514f3%40app.fastmail.com
>>  
>> <https://groups.google.com/d/msgid/sqlalchemy/c5b89f67-e3fa-45f2-8774-463d154514f3%40app.fastmail.com?utm_medium=email_source=footer>.
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/CAEx_o%2BAhhUSPDYuGL4TFkTvxjHYfKEWUnBMxh2PzCDw3GJPaGg%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/sqlalchemy/CAEx_o%2BAhhUSPDYuGL4TFkTvxjHYfKEWUnBMxh2PzCDw3GJPaGg%40mail.gmail.com?utm_medium=email_source=footer>.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/1047d2d1-6964-4d03-a978-dbac1b3e57ec%40app.fastmail.com.

Re: [sqlalchemy] Event listener for when query results get matched with pre-existing objects on the session

[Mike Bayer]

On Fri, Oct 20, 2023, at 8:08 AM, 'Tony Cosentini' via sqlalchemy wrote:
> Hi,
> 
> Is there any way to listen for an event for when a query result gets merged 
> into a pre-existing object in the session?

this is the refresh event:

https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh


> 
> I'm working on disabling autoflush for some of our codebase (mostly to cut 
> down on queries on a high QPS path), but before doing that I want to write a 
> utility to detect when we read data that was flushed via autoflush to cut 
> down on any potential bugs.
> 
> What I'd like to do is this (while autoflush is still enabled):
> 
> table_one = TableOne(name='test')
> session.add(table_one) # Track this object as pending a flush via the 
> before_* listeners, this is working as expected.
> 
> table_two_instances = session.query(TableTwo).all() # All good, doesn't do 
> anything with the table_one instance created earlier
> 
> table_one_instances = session.query(TableOne).all() # I would like to log a 
> warning here as the results of this query depend on a flush happening. What 
> I'm hoping to do is detect that one of the rows coming back is the same 
> object that was flagged earlier, but I can't see to find the correct event to 
> use.

you probably want to also use refresh_flush also, which will be invoked for 
column defaults that are populated on the object

https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh_flush




> 
> Is this possible? I can't seem to find the appropriate event that would 
> trigger when the results from the .all() query get merged back in with the 
> existing objects in the session.

that's the refresh event

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/c5b89f67-e3fa-45f2-8774-463d154514f3%40app.fastmail.com.

Re: [sqlalchemy] Batch patch ORM entities

[Mike Bayer]

On Wed, Oct 11, 2023, at 11:10 AM, Pierre Massé wrote:
> Thanks a lot for the insight Mike,
> 
> My question might then be quite naive: let's say I have a list of ORM 
> entities on one side, and an accordingly sorted numpy array of computed 
> features, how would I merge back attributes on entities?
> 
> Let's say I have a list like :
> user_list = [User(id=1, dist=None), User(id=2, dist=None)]
> 
> and a pandas DataFrame (or numpy array) like:
> 
> dist_df = 
> id dist
> 1 123
> 2 90
> 
> How would I correlate those 2 into:
> [User(id=1, dist=123), User(id=2, dist=90)]
> 
> Would the way to go be a simple for loop? Like:
> for user in user_list:
> user.dist = dist_df.loc[user.id, 'dist']
> 
> Or is there something included in SQLAlchemy for this kind of task?

let's assume the two lists are ordered, which I would strongly recommend.  
merge them with zip:

for user, dist in zip(user_list, dist_df):
user.dist = dist



> 
> Regards,
> 
> Pierre
> 
> Le mer. 11 oct. 2023 à 15:07, Mike Bayer 
>  a écrit :
>> __
>> 
>> 
>> On Wed, Oct 11, 2023, at 4:22 AM, Pierre Massé wrote:
>>> Dear all,
>>> 
>>> I have a requirement that makes me think that I need to "mass patch" some 
>>> ORM objects. However, I am open to any suggestions regarding the way to 
>>> answer my requirements.
>>> 
>>> I have defined an ORM object which represents a user, holding longitude and 
>>> latitude (among other attributes). At some point, I want to query many of 
>>> those users, and send them back holding the geographical distance from a 
>>> certain point (defined by longitude and latitude) along with their other 
>>> data.
>>> 
>>> Computing the distance is computationally heavy, and I noticed that I could 
>>> greatly improve performance by mass computing those distances, using numpy 
>>> for example.
>>> 
>>> My question is: would it be possible to split my flow in 2 : 
>>> - a flow that queries the data that is simply available in the database, as 
>>> ORM entities
>>> - a flow that queries lon/lat as a numpy array, perform the distance 
>>> computation
>>> and afterward merge those 2 in the queried ORM entities?
>> 
>> This is a straightforward programming task.   Query for the set of objects 
>> you want, assemble the appropriate values into a numpy array, do wahtever 
>> numpy thing you need, then merge back.you'd likely want to ensure you 
>> can correlate numpy rows back to original objects most likely by keeping a 
>> sort order between your result set and your numpy array.
>> 
>> not stated here is if these numpy-calculated values as assembled onto ORM 
>> objects are also database-column mapped, it sounds like they are not 
>> (otherwise those values would be in the database), so the matrix values can 
>> be applied to plain attributes on the objects directly.
>> 
>> now where this may be more challenging, not sure if this is what you're 
>> asking, is if you want this to happen implicitly for all queries or 
>> something like that.   there's ways to do this depending on the programming 
>> patterns you are looking to achieve however I'd certainly start simple with 
>> a function like "apply_lat_long_to_list_of_objects(obj)".
>> 
>> 
>>> 
>>> It is important to me that I finally get back a list of ORM entities fully 
>>> populated, because my whole downstream process is built around this 
>>> assumption.
>>> 
>>> Thanks a lot for your insights on the matter!
>>> 
>>> Regards,
>>> 
>>> Pierre
>>> 
>>> PS: giving me a "SQLAlchemy fast distance computation" won't do the trick, 
>>> because I have other kinds of computations that may not be optimizable this 
>>> way.
>>> 
>>> 
>>> --
>>> SQLAlchemy -
>>> The Python SQL Toolkit and Object Relational Mapper
>>>  
>>> http://www.sqlalchemy.org/
>>>  
>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>> description.
>>> ---
>>> You received this message because you are subscribed to the Google Groups 
>>> "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to sqlalchemy+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://grou

Re: [sqlalchemy] Batch patch ORM entities

[Mike Bayer]

On Wed, Oct 11, 2023, at 4:22 AM, Pierre Massé wrote:
> Dear all,
> 
> I have a requirement that makes me think that I need to "mass patch" some ORM 
> objects. However, I am open to any suggestions regarding the way to answer my 
> requirements.
> 
> I have defined an ORM object which represents a user, holding longitude and 
> latitude (among other attributes). At some point, I want to query many of 
> those users, and send them back holding the geographical distance from a 
> certain point (defined by longitude and latitude) along with their other data.
> 
> Computing the distance is computationally heavy, and I noticed that I could 
> greatly improve performance by mass computing those distances, using numpy 
> for example.
> 
> My question is: would it be possible to split my flow in 2 : 
> - a flow that queries the data that is simply available in the database, as 
> ORM entities
> - a flow that queries lon/lat as a numpy array, perform the distance 
> computation
> and afterward merge those 2 in the queried ORM entities?

This is a straightforward programming task.   Query for the set of objects you 
want, assemble the appropriate values into a numpy array, do wahtever numpy 
thing you need, then merge back.you'd likely want to ensure you can 
correlate numpy rows back to original objects most likely by keeping a sort 
order between your result set and your numpy array.

not stated here is if these numpy-calculated values as assembled onto ORM 
objects are also database-column mapped, it sounds like they are not (otherwise 
those values would be in the database), so the matrix values can be applied to 
plain attributes on the objects directly.

now where this may be more challenging, not sure if this is what you're asking, 
is if you want this to happen implicitly for all queries or something like 
that.   there's ways to do this depending on the programming patterns you are 
looking to achieve however I'd certainly start simple with a function like 
"apply_lat_long_to_list_of_objects(obj)".


> 
> It is important to me that I finally get back a list of ORM entities fully 
> populated, because my whole downstream process is built around this 
> assumption.
> 
> Thanks a lot for your insights on the matter!
> 
> Regards,
> 
> Pierre
> 
> PS: giving me a "SQLAlchemy fast distance computation" won't do the trick, 
> because I have other kinds of computations that may not be optimizable this 
> way.
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/CAH4TWVuJWP9WsSYNScPH%2BK9JJ3PqbOwxkm%3D_PXbPtYXzpBdvcg%40mail.gmail.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/f7dcbd42-0509-46c5-b668-1a15a4834cab%40app.fastmail.com.

Re: [sqlalchemy] How to use IBM i Access ODBC Driver

[Mike Bayer]

SQLAlchemy does not include support for this driver - the only IBM driver is 
the ibm_db_sa you reference, so short of writing your own driver based on that 
one, that's what's available.



On Tue, Oct 10, 2023, at 11:07 PM, Jack W. wrote:
> I have the IBM i Access ODBC Driver 
>  
> installed on Linux and I'm trying out Apache Superset.
> 
> The SQLAlchemy URIs I create don't work, e.g.,
>  • iaccess+pyodbc://x  driver not found
>  • All the "traditional" URIs, e.g., using ibm_db_sa+pyodbc:///xxx don't 
> work because I don't have that older driver 
> 
>  installed (which does seem to be supported by SQLAlchemy).
> TIA for any tips.
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/ba319cc9-ef50-4eee-b16c-989a3e4fd971n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/1c282b20-613e-4de6-9f8e-0325706d9911%40app.fastmail.com.

Re: [sqlalchemy] why does sqlalchemy UUID succeed with mysql BINARY(16) but fails with postgres BYTEA

[Mike Bayer]

PostgreSQL has a native UUID datatype, so when you use SQLAlchemy's UUID, it 
maps to a real PG UUID datatype, not BYTEA.

as for sqlalchemy_utils.UUIDType, we dont maintain that package here so you'd 
need to look at their source code.

Overall if you want complete "UUID mapped to any arbitrary binary datatype on 
any platform unambiguously", code it yourself, using the example at 
https://docs.sqlalchemy.org/en/20/core/custom_types.html#backend-agnostic-guid-type
 as a guide.  that example currently uses CHAR(32), but you can change it to 
use BYTEA/BINARY with appropriate changes.   sqlalchemy_utils likely got their 
UUID type from an older version of this example.

On Tue, Oct 10, 2023, at 8:35 PM, Sam S wrote:
> I noticed that mysql's BINARY(16) datatype can get read into sqlalchemy's 
> UUID() no problem, but if it is postgres' BYTEA datatype, then it fails. I 
> created a reproducible example at this gist: 
> https://gist.github.com/sss-ng/1283c85d1010264132985156c148439b
> 
> If someone could help me, I would like to know how i can read a binary UUID 
> into model in postgres using sqlalchemy.
> 
> Thank you
> Sam
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/56e11560-7a36-4a2b-9364-8a5f4201946fn%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/c3a15e41-b9b4-41ea-95db-ab9402fc51f1%40app.fastmail.com.

Re: [sqlalchemy] sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'uuid_utils.UUID'

[Mike Bayer]

I havent read about uuid 7, however if it takes up the same number of bytes as 
other UUIDs, this "uuid_utils" should really be returning Python UUID objects 
and not something ad-hoc, that's the source of the incompatibility here.

otherwise, you'd need to build a custom type, see 
https://docs.sqlalchemy.org/en/20/core/custom_types.html#backend-agnostic-guid-type
 where for PG you'd have to adapt this UUID v7 into a plain Python UUID object, 
which the psycopg dialects can then interpret.

On Mon, Oct 2, 2023, at 6:20 PM, Zer0x00 wrote:
> I want to use UUID v7 with this package:
> https://github.com/aminalaee/uuid-utils/
> 
> Here's my current code:
> from sqlalchemy.dialects.postgresql import UUID as POSTGRES_UUID
> from sqlalchemy.orm import Mapped, as_declarative, mapped_column
> from uuid_utils import UUID, uuid7
> 
> from db.dependencies import get_db
> 
> 
> @as_declarative()
> class Base:
> pass
> 
> 
> class Test(Base):
> __tablename__: str = "test"
> 
> id: Mapped[UUID] = mapped_column(POSTGRES_UUID(as_uuid=True), 
> primary_key=True, default=uuid7)
> 
> 
> a = Test()
> 
> 
> with get_db() as db:
> db.add(a)
> db.commit()
> 
> 
> 
> which throws this error
> 
> Traceback (most recent call last):
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", 
> line 1965, in _exec_single_context
> self.dialect.do_execute(
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", 
> line 921, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.ProgrammingError: can't adapt type 'uuid_utils.UUID'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/dci-ng/test6.py", line 35, in 
> db.commit()
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", 
> line 1923, in commit
> trans.commit(_to_root=True)
>   File "", line 2, in commit
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py",
>  line 139, in _go
> ret_value = fn(self, *arg, **kw)
> 
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", 
> line 1239, in commit
> self._prepare_impl()
>   File "", line 2, in _prepare_impl
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py",
>  line 139, in _go
> ret_value = fn(self, *arg, **kw)
> 
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", 
> line 1214, in _prepare_impl
> self.session.flush()
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", 
> line 4179, in flush
> self._flush(objects)
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", 
> line 4314, in _flush
> with util.safe_reraise():
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py",
>  line 146, in __exit__
> raise exc_value.with_traceback(exc_tb)
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", 
> line 4275, in _flush
> flush_context.execute()
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", 
> line 466, in execute
> rec.execute(self)
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", 
> line 642, in execute
> util.preloaded.orm_persistence.save_obj(
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py",
>  line 93, in save_obj
> _emit_insert_statements(
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py",
>  line 1226, in _emit_insert_statements
> result = connection.execute(
>  ^^^
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", 
> line 1412, in execute
> return meth(
>^
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", 
> line 516, in _execute_on_connection
> return connection._execute_clauseelement(
>^^
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", 
> line 1635, in _execute_clauseelement
> ret = self._execute_context(
>   ^^
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", 
> line 1844, in _execute_context
> return self._exec_single_context(
>^^
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", 
> line 1984, in _exec_single_context
> self._handle_dbapi_exception(
>   File 
> "/home/dev/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", 
> line 2339, in _handle_dbapi_exception
> raise 

Re: [sqlalchemy] Issue DELETE statement with LIMIT

[Mike Bayer]

I thought we have a github for DELETE..LIMIT but we dont.   This would be a 
construct specific to the MySQL dialect:  from sqlalchemy.dialects.mysql import 
delete , where it would include order_by() and limit() params.We don't have 
internal resources to carry this through so we'd rely upon high quality pull 
requests with tests, once the issue is created.

for now I would suggest using text() for the case that an application needs 
this extremely unusual construct once in awhile.

On Fri, Sep 22, 2023, at 7:16 AM, 'Grennith' via sqlalchemy wrote:
> Hi everyone,
> I'd like to issue a LIMIT for a DELETE statement.
> By default, this is not possible as far as I can see it. The function limit() 
> is available for SELECT in general however. 
> Searching through documentation, I found a reference to with_dialect_option():
> https://docs.sqlalchemy.org/en/20/search.html?q=with_dialect_options_keywords=yes=default#
> Which points to 
> https://docs.sqlalchemy.org/en/20/core/dml.html#sqlalchemy.sql.expression.UpdateBase.with_dialect_options
>  claiming the function to be available in UpdateBase (although the 
> documentation off given the claimed method is not callable like documented). 
> This was the case in 1.4 
> (https://github.com/sqlalchemy/sqlalchemy/blob/rel_1_4/lib/sqlalchemy/sql/dml.py#L345)
>  already and also in 2.0 
> (https://github.com/sqlalchemy/sqlalchemy/blob/main/lib/sqlalchemy/sql/dml.py#L438).
> 
> However, trying to call it as per documentation results in an exception being 
> raised:
> ```
> sqlalchemy.exc.ArgumentError: Argument 'mysql_limit' is not accepted by 
> dialect 'mysql' on behalf of 
> ```
> This is caused by 
> https://github.com/sqlalchemy/sqlalchemy/blob/main/lib/sqlalchemy/dialects/mysql/base.py#L2454
>  not listing sql.Delete explicitly. UpdateBase apparently cannot be 
> referenced either given the import (guessing as it's not imported explicitly 
> in 
> https://github.com/sqlalchemy/sqlalchemy/blob/main/lib/sqlalchemy/sql/__init__.py
>  or not referenced another way?).
> However, by adding Delete just like Update will have the following line run 
> fine without an error - but not adding the LIMIT either. My best guess right 
> now would be due to the lack of limit clause handling?
> ```
> stmt = stmt.with_dialect_options(mysql_limit=limit, mariadb_limit=limit)
> ```
> where `limit` simply is an integer.
> 
> 
> Any hints or help is appreciated. I can also raise a ticket on Github :)
> 
> Best regards
> 
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/f13a8ca6-3e52-4287-a6a4-52b5b4672470n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/ae24d467-9750-496a-a3b7-1da85afc70f2%40app.fastmail.com.

Re: [sqlalchemy] Event: do_orm_execute inserted value

[Mike Bayer]

for a statement like that where the values are embedded in the insert() 
construct directly you would use:

 orm_execute_state.statement.compile().params

this is mentioned in the tutorial at 
https://docs.sqlalchemy.org/en/20/tutorial/data_insert.html#the-insert-sql-expression-construct
 

otherwise if parameters are sent separately, they are in 
orm_execute_state.parameters



On Wed, Sep 20, 2023, at 8:37 AM, Tomas Pavlovsky wrote:
> Hi Mike,
> 
> how to access inserted value in do_orm_execute? 
> Event "before_insert" has "target" parameter, but how to do it in 
> do_orm_execute?
> session.execute(insert(User).values(name="name"))
> I need to access name value but don't know how?
> Thanks
> 
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/9bea7c27-f67b-4d87-ab5e-71ece8c1dd46n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/7aa3f480-6266-437d-aab6-d2e1c3413f79%40app.fastmail.com.

Re: [sqlalchemy] Temporarily disable/intercept ORM events on mutation

[Mike Bayer]

On Fri, Sep 15, 2023, at 8:59 PM, Mike Bayer wrote:
> 
> unfortunately no, that's a backref event handler, that's within the class 
> instrumentation and has no mechanism to be disabled on a per-class basis, not 
> to mention the backref handler is not the only aspect of things that expects 
> a certain kind of value to be present.  Marshmallow should have alternate 
> APIs that allow the attributes in question to be named by their string name 
> somewhere, rather than being assigned some out-of-band object; that's not 
> very good design in today's pep-484 typed Python.

sorry, I meant "no mechanism to be disabled on a per-instance basis". 

> 
> 
> 
> 
> 
>> 
>> 
>> --
>> SQLAlchemy -
>> The Python SQL Toolkit and Object Relational Mapper
>>  
>> http://www.sqlalchemy.org/
>>  
>> To post example code, please provide an MCVE: Minimal, Complete, and 
>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>> description.
>> ---
>> You received this message because you are subscribed to the Google Groups 
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to sqlalchemy+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/sqlalchemy/34ee3121-e284-4fcb-9c1d-6191557e89bcn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/sqlalchemy/34ee3121-e284-4fcb-9c1d-6191557e89bcn%40googlegroups.com?utm_medium=email_source=footer>.
> 

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/0183d49d-e442-4512-bf62-8adae2c0e903%40app.fastmail.com.

Re: [sqlalchemy] Temporarily disable/intercept ORM events on mutation

[Mike Bayer]

On Fri, Sep 15, 2023, at 4:28 PM, 'Luna Lucadou' via sqlalchemy wrote:
> When customers call our JSON:API API, they can use an "include" parameter to 
> specify related objects to be appended to the response.
> 
> However, requests to include are not always satisfiable (e.g. if 
> job.supervisor=null, include=supervisor is ignored).
> In order to prevent Marshmallow from trying to load nonexistent related 
> objects to append to our API responses, we need to tell it when to ignore a 
> relationship attribute, such as via setting 
> job.supervisor=marshmallow.missing (if it sees job.supervisor=None, it will 
> attempt (and fail) to load the null supervisor object, so we cannot just 
> leave it as-is).
> 
> Unfortunately, this causes problems as SQLAlchemy attempts to handle the new 
> value:
> 
> Error Traceback (most recent call last): File 
> "/Users/lucadou/IdeaProjects/person-api/api/unit_tests/test_person_service.py",
>  line 601, in test_get_people_include_job response = 
> self.person_service.get_people(QueryParameters({"include": "jobs"})) 
>  File 
> "/Users/lucadou/IdeaProjects/person-api/api/src/person_service.py", line 61, 
> in get_people response = person_schema.dump(people, many=True) 
> ^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
>  line 557, in dump result = self._serialize(processed_obj, many=many) 
> ^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
>  line 519, in _serialize return [ ^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
>  line 520, in  self._serialize(d, many=False) File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
>  line 525, in _serialize value = field_obj.serialize(attr_name, obj, 
> accessor=self.get_attribute) 
>  File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow_jsonapi/fields.py",
>  line 248, in serialize return super().serialize(attr, obj, accessor) 
> ^^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/fields.py",
>  line 344, in serialize return self._serialize(value, attr, obj, **kwargs) 
> ^^^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow_jsonapi/fields.py",
>  line 274, in _serialize self._serialize_included(item) File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow_jsonapi/fields.py",
>  line 280, in _serialize_included result = self.schema.dump(value) 
> ^^^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
>  line 551, in dump processed_obj = self._invoke_dump_processors( 
> ^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
>  line 1068, in _invoke_dump_processors data = self._invoke_processors( 
>  File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/marshmallow/schema.py",
>  line 1225, in _invoke_processors data = processor(data, many=many, **kwargs) 
>  File 
> "/Users/lucadou/IdeaProjects/person-api/api/src/model/schema/job_schema.py", 
> line 62, in set_null_supervisor job.supervisor = missing ^^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
>  line 536, in __set__ self.impl.set( File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
>  line 1466, in set value = self.fire_replace_event(state, dict_, value, old, 
> initiator)  File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
>  line 1505, in fire_replace_event value = fn( ^^^ File 
> "/Users/lucadou/IdeaProjects/person-api/api/venv/lib/python3.11/site-packages/sqlalchemy/orm/attributes.py",
>  line 2167, in emit_backref_from_scalar_set_event instance_state(child), 
> ^ AttributeError: '_Missing' object has no attribute 
> '_sa_instance_state'
> 
> Is there any way to temporarily disable ORM event listeners when we mutate 
> objects and have no intention of saving the changes/do not intend for the ORM 
> to act on them?

unfortunately no, that's a backref event handler, that's within the class 
instrumentation and has no mechanism to be 

Re: [sqlalchemy] How to combine statement eager loading with polymorphic relations

[Mike Bayer]

the recursion_depth feature on selectinload() is also very new and that is 
actually a very complex and not very mature feature.If you only wanted to 
apply additional options for N levels deep, you would build out separate 
options for that, like:

options(
   selectinload(Model.thing, recursion_depth=-1),
   
defaultload(Model.thing).selectinload(Model.otherthing).selectinload(Model.otherthing)
)

something like that

again, fairly esoteric stuff

On Fri, Sep 15, 2023, at 4:25 AM, Cornelis Poppema wrote:
> Hello Mike, thank you very much for the in-depth reply and providing a 
> solution :)
> 
> There is no specific reason that I posted in google groups, I did not visit 
> the support page, this was just something I decided after thinking where to 
> post first between googlegroups and stackoverflow, I will use github 
> discussions when I open a new discussion in the future!
> Your workaround will help me continue so many many thanks. I thought I tried 
> all variations, but I can only concede after seeing this work that I did not 
> attempt adding the selectinload to the root options().
> 
> It might warrant a new thread, but can you also tell me if there is a way to 
> control the options() that sqlalchemy uses for the recursion of next_step ?
> 
> Ie. I included this bit in the original post:
> ```
> .options(
> selectinload(StepModel.next_step, recursion_depth=-1),
> ...
> )
> ```
> 
> and it would be helpful to know if I can chain the same options to next_step, 
> so that step also has its actions and their relationships() eagerly available 
> etc.
> 
> PS
> FWIW the reason I included `raiseload("*")` in options() is because I am 
> running async queries, and personally the error I will be confronted with 
> trying to access lazy attributes is more helpful, so I've come to add it by 
> default. Without raisedload(*) I would see:
> ```
> sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't 
> call await_only() here. Was IO attempted in an unexpected place? (Background 
> on this error at: https://sqlalche.me/e/20/xd2s)
> ```
> 
> With the help of raisedload(*) I get to see:
> ```
> sqlalchemy.exc.InvalidRequestError: 'ServiceActionModel.service' is not 
> available due to lazy='raise```
> ```
> 
> This helps me tackle those cases more easily one-by-one.
> 
> On Thursday, 14 September 2023 at 15:30:23 UTC+2 Mike Bayer wrote:
>> __
>> 
>> 
>> On Thu, Sep 14, 2023, at 7:36 AM, Cornelis Poppema wrote:
>>> Hi all,
>>> 
>>> I am new to sqlalchemy, I think the idea of what I am trying to achieve is 
>>> relatively simple, but I can't seem to figure out how to retrieve 
>>> `.service` in the same query. I failed to find an example in the 2.0 
>>> documentation for exactly this.
>>> 
>>> My attempts have been to simply chain a .selectinload after the 
>>> .selectin_polymorphic, ie.:
>>> 
>>> ```python
>>> .options(
>>> selectinload(StepModel.next_step, recursion_depth=-1),
>>> selectinload(StepModel.actionbases).selectin_polymorphic(
>>> [
>>> ServiceActionModel,
>>> ],
>>> )
>>> .selectinload(ServiceActionModel.service),
>>> raiseload("*"),
>>> )
>>> ```
>>> 
>>> This gives the error:
>>> 
>>> ```
>>> /usr/local/lib/python3.11/site-packages/sqlalchemy/orm/strategy_options.py:2442:
>>>  in _raise_for_does_not_link
>>> raise sa_exc.ArgumentError(
>>> E   sqlalchemy.exc.ArgumentError: ORM mapped entity or attribute 
>>> "ServiceActionModel.service" does not link from relationship 
>>> "StepModel.actionbases".  Did you mean to use 
>>> "StepModel.actionbases.of_type(ServiceActionModel)"?
>>> ```
>>> 
>> 
>> Hi -
>> 
>> A few up front things, is it possible you can post these questions that have 
>> a lot of code on github discussions?   That's at 
>> https://github.com/sqlalchemy/sqlalchemy/discussions .I'm sort of 
>> wondering how new users are still arriving here at the mailing list, did you 
>> find this list via the support page at 
>> https://www.sqlalchemy.org/support.html ?   I would want to change the 
>> verbiage there to please refer people to GH discussions instead.
>> Especially with these advanced inheritance eager loading problems, which in 
>> the majority of cases end up being real bugs in SQLAlchemy, as seems to be 
>> the c

Re: [sqlalchemy] How to combine statement eager loading with polymorphic relations

[Mike Bayer]

working on that issue but you should also be able to do this right now:

.options(

selectinload(StepModel.actionbases.of_type(ServiceActionModel)).selectinload(ServiceActionModel.service),
raiseload("*"),
)

that produces more of a LEFT OUTER JOIN with a subquery situation but still 
"works"

On Thu, Sep 14, 2023, at 7:36 AM, Cornelis Poppema wrote:
> Hi all,
> 
> I am struggling to combine a joinedload (or selectinload, whatever works) 
> with the submodels of selectin_polymorphic.
> 
> I have a model "step" that has a collections of "actions" that are relevant 
> when my application reaches that step. These actions can be anything and also 
> have their own relationships to other models that I want to eagerly load, all 
> while querying "step". I would strongly prefer to achieve this in a query 
> statement over defining eager loading in the relationship declarations on the 
> models itself.
> 
> Here are my models:
> 
> ```python
> from enum import StrEnum, auto
> from sqlalchemy import Column, Enum, ForeignKey, Integer, String
> from sqlalchemy.dialects.postgresql import UUID
> from sqlalchemy.orm import as_declarative, declared_attr, relationship
> 
> 
> class ActionTypeEnum(StrEnum):
> flow = auto()
> service = auto()
> transition = auto()
> 
> 
> @as_declarative()
> class BaseSqlModel:
> pk = Column(Integer, primary_key=True, index=True)
> 
> 
> class IdColumnMixin:
> @declared_attr
> def id(cls):
> return Column(UUID(as_uuid=True), unique=True, nullable=False, 
> index=True)
> 
> 
> class StepModel(IdColumnMixin, BaseSqlModel):
> __tablename__ = "step"
> next_step_id = Column(ForeignKey("step.id", use_alter=True))
> next_step = relationship("StepModel", remote_side="StepModel.id")
> 
> actionbases = relationship("ActionBaseModel")
> 
> 
> class ActionBaseModel(IdColumnMixin, BaseSqlModel):
> __tablename__ = "actionbase"
> action_type = Column(Enum(ActionTypeEnum), nullable=False)
> step_id = Column(ForeignKey("step.id"), nullable=False)
> step = relationship("StepModel", back_populates="actionbases")
> 
> __mapper_args__ = {
> "polymorphic_identity": "actionbase",
> "polymorphic_on": "action_type",
> }
> 
> 
> class ServiceModel(IdColumnMixin, BaseSqlModel):
> __tablename__ = "service"
> name = Column(String(200), nullable=False)
> 
> 
> class ServiceActionModel(ActionBaseModel):
> __tablename__ = "serviceaction"
> id = Column(ForeignKey("actionbase.id"), primary_key=True)
> service_id = Column(ForeignKey("service.id"), nullable=True)
> service = relationship("ServiceModel")
> 
> __mapper_args__ = {
> "polymorphic_identity": ActionTypeEnum.service,
> }
> ```
> 
> To query step I write this:
> 
> ```python
> db_step = (
> await self.session.execute(
> select(StepModel)
> .filter_by(id=id)
> .options(
> selectinload(StepModel.next_step, recursion_depth=-1),
> selectinload(StepModel.actionbases).selectin_polymorphic(
> [
> ServiceActionModel,
> ],
> ),
> raiseload("*"),
> )
> )
> ).scalar_one()
> ```
> 
> Accessing `db_step.actionbases[0]` works as expected: it is of type 
> ServiceActionModel, accessing `db_step.actionbases[0].service.name` throws 
> the expected error:
> ```
> /usr/local/lib/python3.11/site-packages/sqlalchemy/orm/strategies.py:862: in 
> _invoke_raise_load
> raise sa_exc.InvalidRequestError(
> E   sqlalchemy.exc.InvalidRequestError: 'ServiceActionModel.service' is not 
> available due to lazy='raise'
> ```
> 
> I am new to sqlalchemy, I think the idea of what I am trying to achieve is 
> relatively simple, but I can't seem to figure out how to retrieve `.service` 
> in the same query. I failed to find an example in the 2.0 documentation for 
> exactly this.
> 
> My attempts have been to simply chain a .selectinload after the 
> .selectin_polymorphic, ie.:
> 
> ```python
> .options(
> selectinload(StepModel.next_step, recursion_depth=-1),
> selectinload(StepModel.actionbases).selectin_polymorphic(
> [
> ServiceActionModel,
> ],
> )
> .selectinload(ServiceActionModel.service),
> raiseload("*"),
> )
> ```
> 
> This gives the error:
> 
> ```
> /usr/local/lib/python3.11/site-packages/sqlalchemy/orm/strategy_options.py:2442:
>  in _raise_for_does_not_link
> raise sa_exc.ArgumentError(
> E   sqlalchemy.exc.ArgumentError: ORM mapped entity or attribute 
> "ServiceActionModel.service" does not link from relationship 
> "StepModel.actionbases".  Did you mean to use 
> "StepModel.actionbases.of_type(ServiceActionModel)"?
> ```
> 
> Which seems fair; there is no relationship defined on ServiceActionModel to 
> StepModel. (but there is on ActionBaseModel).
> 
> So I've tried part 2 of 

Re: [sqlalchemy] How to combine statement eager loading with polymorphic relations

[Mike Bayer]

On Thu, Sep 14, 2023, at 7:36 AM, Cornelis Poppema wrote:
> Hi all,
> 
> I am new to sqlalchemy, I think the idea of what I am trying to achieve is 
> relatively simple, but I can't seem to figure out how to retrieve `.service` 
> in the same query. I failed to find an example in the 2.0 documentation for 
> exactly this.
> 
> My attempts have been to simply chain a .selectinload after the 
> .selectin_polymorphic, ie.:
> 
> ```python
> .options(
> selectinload(StepModel.next_step, recursion_depth=-1),
> selectinload(StepModel.actionbases).selectin_polymorphic(
> [
> ServiceActionModel,
> ],
> )
> .selectinload(ServiceActionModel.service),
> raiseload("*"),
> )
> ```
> 
> This gives the error:
> 
> ```
> /usr/local/lib/python3.11/site-packages/sqlalchemy/orm/strategy_options.py:2442:
>  in _raise_for_does_not_link
> raise sa_exc.ArgumentError(
> E   sqlalchemy.exc.ArgumentError: ORM mapped entity or attribute 
> "ServiceActionModel.service" does not link from relationship 
> "StepModel.actionbases".  Did you mean to use 
> "StepModel.actionbases.of_type(ServiceActionModel)"?
> ```

Hi -

A few up front things, is it possible you can post these questions that have a 
lot of code on github discussions?   That's at 
https://github.com/sqlalchemy/sqlalchemy/discussions .I'm sort of wondering 
how new users are still arriving here at the mailing list, did you find this 
list via the support page at https://www.sqlalchemy.org/support.html ?   I 
would want to change the verbiage there to please refer people to GH 
discussions instead.Especially with these advanced inheritance eager 
loading problems, which in the majority of cases end up being real bugs in 
SQLAlchemy, as seems to be the case here (at least, there is an inconsistency 
in the API that somehow needs to be documented, or something).

As for the question, first off this is really advanced usage and I've hardly 
ever seen people using selectin_polymorphic(), much less deep within a chain of 
loaders like this.

The correct form for this load would follow from how it's described at 
https://docs.sqlalchemy.org/en/20/orm/queryguide/inheritance.html#combining-additional-loader-options-with-selectin-polymorphic-subclass-loads
 , where the ORM allows the selectin_polymorphic(Target, [TargetSubclassA]) to 
be a sibling to the appropriate relationship load, 
selectinload(TargetSubclassA.elements).   The example there places both of 
these options comma-separated within select().options().This is the 
"inconsistent" part because I'm already surprised the ORM is allowing the 
selectinload() to be present against TargetSubclassA when that's not one of the 
primary entities in the select().

However in your case, you are coming off of a parent loader option already.  So 
following from this, the correct form based on a direct reading of those docs 
would, *in theory*, be:

select().options(
 selectinload(Parent.target).options(
 selectin_polymorphic(Target, [TargetSubclassA]),
 selectinload(TargetSubclassA.elements)
 )
)

that is, you can build up sibling options from a parent loader option using 
another call to .options().

however, this doesn't work; the validation of the loader chain unsurprisingly 
notes that TargetSubclass is not linked from Parent.target, and they'd like you 
to use of_type() instead.So I've made a bug for this here:  
https://github.com/sqlalchemy/sqlalchemy/issues/10348  as something has to 
change here, either the docs, or the usage pattern for selectin_polymorphic(), 
or the error checks have to get a lot smarter to figure this out and let this 
case pass, since it works fine the way you are expecting if I just have it skip 
the error checking.

What you can do now is use with_polymorphic() instead that is more configurable 
for this kind of loader chain:

TT = with_polymorphic(Target, [TargetSubclassA])
select().options(selectinload(Parent.target.of_type(TT).selectinload(TT.TargetSubclassA.elements)))

discussion can continue at the issue above or a new github discussion, thanks!

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/a99d9853-8936-455f-9439-bfe5f2e03faa%40app.fastmail.com.

Re: [sqlalchemy] Unique constraint error in PostgrSQL when migrating with Alembic

[Mike Bayer]

your revisions table has a composite unique constraint - one constraint with 
two columns in it.   therefore to refer to this constraint via foreign key, you 
need a single composite foreignkeyconstraint - again, one constraint that links 
two columns together.  you would use ForeignKeyConstraint for this, not 
ForeignKey, see ForeignKeyConstraint in 
https://docs.sqlalchemy.org/en/20/core/constraints.html#defining-foreign-keys .

On Wed, Sep 13, 2023, at 1:43 PM, Lord Wolfenstein wrote:
> I have a database that looks like this that I create with Alembic. The 
> relationships are trivial except between Revision and ObjectCount, there it 
> used two foreign keys
> aaa.png
> 
> The code looks like this 
> 
> 
> 
> from datetime import datetime
> from typing import Optional
> 
> from sqlalchemy import ForeignKey, create_engine, UniqueConstraint
> from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, 
> relationship, sessionmaker  # type: ignore
> 
> 
> class Base(DeclarativeBase):
> pass
> 
> 
> class Blueprint(Base):
> __tablename__ = "blueprints"
> 
> blueprint_id: Mapped[int] = mapped_column(primary_key=True)
> filename: Mapped[str]
> created: Mapped[datetime] = mapped_column(default=datetime.utcnow)
> 
> revisions: Mapped[list["Revision"]] = 
> relationship(back_populates="blueprint")  # type: ignore
> 
> def get_new_revision_number(self) -> int:
> if not self.revisions:
> return 1
> return max(revision.revision_number for revision in self.revisions) + 
> 1
> 
> def __str__(self):
> return f"{self.filename} : {self.blueprint_id}"
> 
> 
> class Revision(Base):
> __tablename__ = "revisions"
> __table_args__ = (
> UniqueConstraint("blueprint_id", "revision_number", 
> name="revision_blueprint_revision_number"),
> )
> 
> blueprint_id: Mapped[int] = 
> mapped_column(ForeignKey("blueprints.blueprint_id"), primary_key=True)
> revision_number: Mapped[int] = mapped_column(primary_key=True)
> date: Mapped[datetime] = mapped_column(default=datetime.utcnow)
> savedata: Mapped[str]
> 
> blueprint: Mapped["Blueprint"] = relationship(back_populates="revisions") 
>  # type: ignore
> object_count: Mapped[Optional[list["ObjectCount"]]] = relationship(
> back_populates="revision",
> primaryjoin="and_(Revision.blueprint_id==ObjectCount.blueprint_id, 
> Revision.revision_number==ObjectCount.revision_number)",
> )  # type: ignore
> 
> def __str__(self):
> return f"{self.blueprint.filename} : {self.blueprint_id} : 
> {self.revision_number}"
> 
> 
> class Object(Base):
> __tablename__ = "objects"
> 
> object_id: Mapped[int] = mapped_column(primary_key=True)
> name: Mapped[str]
> 
> def __str__(self):
> return self.name
> 
> 
> # https://docs.sqlalchemy.org/en/20/orm/join_conditions.html
> # https://docs.sqlalchemy.org/en/20/orm/basic_relationships.html
> # 
> https://docs.sqlalchemy.org/en/20/core/constraints.html#sqlalchemy.schema.UniqueConstraint
> class ObjectCount(Base):
> __tablename__ = "object_count"
> __table_args__ = (
> UniqueConstraint("blueprint_id", "revision_number", "object_id", 
> name="o_c_unique"),
> )
> 
> blueprint_id: Mapped[int] = 
> mapped_column(ForeignKey("revisions.blueprint_id"), primary_key=True)
> revision_number: Mapped[int] = 
> mapped_column(ForeignKey("revisions.revision_number"), primary_key=True)
> object_id: Mapped[int] = mapped_column(ForeignKey("objects.object_id"), 
> primary_key=True)
> 
> count: Mapped[int]
> 
> object: Mapped["Object"] = relationship()  # type: ignore
> revision: Mapped["Revision"] = relationship(
> back_populates="object_count",
> primaryjoin="and_(Revision.blueprint_id==ObjectCount.blueprint_id, 
> Revision.revision_number==ObjectCount.revision_number)",
> )  # type: ignore
> 
> def __str__(self):
> return f"{self.revision.blueprint.filename} {self.revision_number} 
> {self.object.name} {self.count}"
> 
> 
> DATABASE = "postgresql+psycopg://user:password@192.168.10.111:5432/mydatabase"
> DATABASE = 
> "postgresql+psycopg2://user:password@192.168.10.111:5432/mydatabase"
> #DATABASE = "sqlite:///database.db"
> engine = create_engine(DATABASE)
> SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
> 
> 
> I can create migrations with Alembic no problem. And I can migrate when I use 
> SQLite. But when I try to migrate with PostgreSQL I get this error:
> 
> sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InvalidForeignKey) there is 
> no unique constraint matching given keys for referenced table "revisions"
> 
> [SQL:
> CREATE TABLE object_count (
> blueprint_id INTEGER NOT NULL,
> revision_number INTEGER NOT NULL,
> object_id INTEGER NOT NULL,
> count INTEGER NOT NULL,
> PRIMARY KEY (blueprint_id, 

Re: [sqlalchemy] Update Username and Password from vault

[Mike Bayer]

makes sense, the connection is pooled.  if you make lots of connections, like 
more than five simultaneous connections, you'd see more of it, if you call 
engine.dispose() then engine.connect(), you would see it again also, etc.   
Also try using NullPool, then you'd see the hook run every time the engine is 
used.

On Fri, Sep 8, 2023, at 12:13 PM, Steven Schierholz wrote:
> Yes but only once when the app starts up.
> 
> On Friday, September 8, 2023 at 10:04:45 AM UTC-6 Mike Bayer wrote:
>> __
>> assuming proper indentation it looks fine.  are your print statements being 
>> seen ?
>> 
>> On Fri, Sep 8, 2023, at 11:54 AM, Steven Schierholz wrote:
>>> Ok that makes sense and clarifies some stuff for me. I have tried your 
>>> implementation but it doesn't seem like its getting new connections. We are 
>>> using sessionmaker(). So basically this is what we are doing. Can you help 
>>> me understand if we are doing this right and if any changes need to happen 
>>> to make this work? Sorry the tabbing is not right after paste. Thanks for 
>>> your help!
>>> 
>>> 
>>> *# Create the engine to connect to the database
*engine = create_engine(
>>>   f"postgresql+psycopg2://test:password@{pg_host}:5432/{pg_database}",
>>> *  # connect_args=ssl_args,
***  connect_args={"options": "-c timezone=utc"},
>>>   pool_pre_ping=*True*,
>>>   encoding="utf8",
>>> )
>>> 
>>> @event.listens_for(engine, "do_connect")
>>> *def *receive_do_connect(dialect, conn_rec, cargs, cparams):
>>> 
>>> *# Getting the postgres details
** **try*:
>>> *# Get the configs
** *configs = Properties()
>>> 
>>> *# Open the file to get the values needed
** **with **open*("/var/secrets/pgsql/vault-credentials.properties", "rb") *as 
*config_file:
>>> configs.load(config_file)
>>> 
>>> *# Get each of the properties, hopefully
** *pg_user = configs.get("username").data
>>> pg_password = configs.get("password").data
>>> 
>>> *except **FileNotFoundError*:
>>> 
>>> *# Use whats in the environment
** *pg_user = os.getenv("pg_user")
>>> pg_password = os.getenv("pg_password")
>>> 
>>> *print*("Connecting to db with username: ", pg_user)
>>> *print*("Connecting to db with password: ", pg_password)
>>> 
>>> cparams["user"] = pg_user
>>> cparams["password"] = pg_password
>>> 
>>> session_factory = sessionmaker(bind=engine)
>>> sqla_session = session_factory()
>>> 
>>> # Then using the sqla_session to execute queries and make modifications to 
>>> the database
>>> On Thursday, September 7, 2023 at 4:26:05 PM UTC-6 Mike Bayer wrote:
>>>> __
>>>> no, create_engine() does not connect at all. connections occur when 
>>>> you first call `engine.connect()`.   From that point, the behavior of 
>>>> subsequent `engine.connect()` calls depends on connection pool 
>>>> configuration.   all connection pools have points at which they continue 
>>>> to establish new connections as the application proceeds, it's just a 
>>>> question of how often and under what circumstances.The default 
>>>> QueuePool will make new connections when it goes into "overflow", as well 
>>>> as when existing connections are invalidated due to connectivity problems 
>>>> or if the pool_recycle timeout is reached.
>>>> 
>>>> 
>>>> 
>>>> On Thu, Sep 7, 2023, at 2:34 PM, Steven Schierholz wrote:
>>>>> That makes sense but doesn't connect only happen once when 
>>>>> create_engine() is called?
>>>>> 
>>>>> On Thursday, September 7, 2023 at 12:00:35 PM UTC-6 Mike Bayer wrote:
>>>>>> __
>>>>>> the documentation for this pattern is at 
>>>>>> https://docs.sqlalchemy.org/en/20/core/engines.html#generating-dynamic-authentication-tokens
>>>>>>  , and a completely specific example is at 
>>>>>> https://docs.sqlalchemy.org/en/20/dialects/mssql.html#mssql-pyodbc-access-tokens
>>>>>>  .   Basically your application needs to have some way to retrieve the 
>>>>>> correct credentials as it runs, and you hook that into the event to 
>>>>>> populate the connect arguments with the correct credentials.
>>>>>> 
>>>>>> On Thu

Re: [sqlalchemy] Update Username and Password from vault

[Mike Bayer]

assuming proper indentation it looks fine.  are your print statements being 
seen ?

On Fri, Sep 8, 2023, at 11:54 AM, Steven Schierholz wrote:
> Ok that makes sense and clarifies some stuff for me. I have tried your 
> implementation but it doesn't seem like its getting new connections. We are 
> using sessionmaker(). So basically this is what we are doing. Can you help me 
> understand if we are doing this right and if any changes need to happen to 
> make this work? Sorry the tabbing is not right after paste. Thanks for your 
> help!
> 
> 
> *# Create the engine to connect to the database
*engine = create_engine(
>   f"postgresql+psycopg2://test:password@{pg_host}:5432/{pg_database}",
> *  # connect_args=ssl_args,
***  connect_args={"options": "-c timezone=utc"},
>   pool_pre_ping=*True*,
>   encoding="utf8",
> )
> 
> @event.listens_for(engine, "do_connect")
> *def *receive_do_connect(dialect, conn_rec, cargs, cparams):
> 
> *# Getting the postgres details
** **try*:
> *# Get the configs
** *configs = Properties()
> 
> *# Open the file to get the values needed
** **with **open*("/var/secrets/pgsql/vault-credentials.properties", "rb") *as 
*config_file:
> configs.load(config_file)
> 
> *# Get each of the properties, hopefully
** *pg_user = configs.get("username").data
> pg_password = configs.get("password").data
> 
> *except **FileNotFoundError*:
> 
> *# Use whats in the environment
** *pg_user = os.getenv("pg_user")
> pg_password = os.getenv("pg_password")
> 
> *print*("Connecting to db with username: ", pg_user)
> *print*("Connecting to db with password: ", pg_password)
> 
> cparams["user"] = pg_user
> cparams["password"] = pg_password
> 
> session_factory = sessionmaker(bind=engine)
> sqla_session = session_factory()
> 
> # Then using the sqla_session to execute queries and make modifications to 
> the database
> On Thursday, September 7, 2023 at 4:26:05 PM UTC-6 Mike Bayer wrote:
>> __
>> no, create_engine() does not connect at all. connections occur when you 
>> first call `engine.connect()`.   From that point, the behavior of subsequent 
>> `engine.connect()` calls depends on connection pool configuration.   all 
>> connection pools have points at which they continue to establish new 
>> connections as the application proceeds, it's just a question of how often 
>> and under what circumstances.The default QueuePool will make new 
>> connections when it goes into "overflow", as well as when existing 
>> connections are invalidated due to connectivity problems or if the 
>> pool_recycle timeout is reached.
>> 
>> 
>> 
>> On Thu, Sep 7, 2023, at 2:34 PM, Steven Schierholz wrote:
>>> That makes sense but doesn't connect only happen once when create_engine() 
>>> is called?
>>> 
>>> On Thursday, September 7, 2023 at 12:00:35 PM UTC-6 Mike Bayer wrote:
>>>> __
>>>> the documentation for this pattern is at 
>>>> https://docs.sqlalchemy.org/en/20/core/engines.html#generating-dynamic-authentication-tokens
>>>>  , and a completely specific example is at 
>>>> https://docs.sqlalchemy.org/en/20/dialects/mssql.html#mssql-pyodbc-access-tokens
>>>>  .   Basically your application needs to have some way to retrieve the 
>>>> correct credentials as it runs, and you hook that into the event to 
>>>> populate the connect arguments with the correct credentials.
>>>> 
>>>> On Thu, Sep 7, 2023, at 1:54 PM, Steven Schierholz wrote:
>>>>> So I have seen some chats here about cred refresh from vault and some 
>>>>> suggestions have been to use @event.listens_for(engine, "do_connect") to 
>>>>> update creds when the connection is established. My understanding of this 
>>>>> is that connecting to the database should only happen once when my flask 
>>>>> application starts up, but I need to update the creds without restarting 
>>>>> my application so I'm not sure that the event listener will work in my 
>>>>> case.
>>>>> 
>>>>> Am I understanding that correctly? If so, is there a way to get the right 
>>>>> creds to pass to the engine for sqlalchemy every 24 hours when the creds 
>>>>> from vault get updated without restarting my application?
>>>>> 
>>>>> 
>>>>> --
>>>>> SQLAlchemy -
>>>>> The Python SQL Toolkit and Object Relati

Re: [sqlalchemy] Update Username and Password from vault

[Mike Bayer]

no, create_engine() does not connect at all. connections occur when you 
first call `engine.connect()`.   From that point, the behavior of subsequent 
`engine.connect()` calls depends on connection pool configuration.   all 
connection pools have points at which they continue to establish new 
connections as the application proceeds, it's just a question of how often and 
under what circumstances.The default QueuePool will make new connections 
when it goes into "overflow", as well as when existing connections are 
invalidated due to connectivity problems or if the pool_recycle timeout is 
reached.



On Thu, Sep 7, 2023, at 2:34 PM, Steven Schierholz wrote:
> That makes sense but doesn't connect only happen once when create_engine() is 
> called?
> 
> On Thursday, September 7, 2023 at 12:00:35 PM UTC-6 Mike Bayer wrote:
>> __
>> the documentation for this pattern is at 
>> https://docs.sqlalchemy.org/en/20/core/engines.html#generating-dynamic-authentication-tokens
>>  , and a completely specific example is at 
>> https://docs.sqlalchemy.org/en/20/dialects/mssql.html#mssql-pyodbc-access-tokens
>>  .   Basically your application needs to have some way to retrieve the 
>> correct credentials as it runs, and you hook that into the event to populate 
>> the connect arguments with the correct credentials.
>> 
>> On Thu, Sep 7, 2023, at 1:54 PM, Steven Schierholz wrote:
>>> So I have seen some chats here about cred refresh from vault and some 
>>> suggestions have been to use @event.listens_for(engine, "do_connect") to 
>>> update creds when the connection is established. My understanding of this 
>>> is that connecting to the database should only happen once when my flask 
>>> application starts up, but I need to update the creds without restarting my 
>>> application so I'm not sure that the event listener will work in my case.
>>> 
>>> Am I understanding that correctly? If so, is there a way to get the right 
>>> creds to pass to the engine for sqlalchemy every 24 hours when the creds 
>>> from vault get updated without restarting my application?
>>> 
>>> 
>>> --
>>> SQLAlchemy -
>>> The Python SQL Toolkit and Object Relational Mapper
>>>  
>>> http://www.sqlalchemy.org/
>>>  
>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>> description.
>>> ---
>>> You received this message because you are subscribed to the Google Groups 
>>> "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to sqlalchemy+...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/sqlalchemy/2de2e3fb-526c-49f7-8780-9ef55a9ad8bcn%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/sqlalchemy/2de2e3fb-526c-49f7-8780-9ef55a9ad8bcn%40googlegroups.com?utm_medium=email_source=footer>.
>> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/5c9c5b5f-a84b-41b8-8eb2-c3117fbf50f5n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/sqlalchemy/5c9c5b5f-a84b-41b8-8eb2-c3117fbf50f5n%40googlegroups.com?utm_medium=email_source=footer>.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/71dcdb8d-2493-403d-83f7-7060041a563c%40app.fastmail.com.

Re: [sqlalchemy] Update Username and Password from vault

[Mike Bayer]

the documentation for this pattern is at 
https://docs.sqlalchemy.org/en/20/core/engines.html#generating-dynamic-authentication-tokens
 , and a completely specific example is at 
https://docs.sqlalchemy.org/en/20/dialects/mssql.html#mssql-pyodbc-access-tokens
 .   Basically your application needs to have some way to retrieve the correct 
credentials as it runs, and you hook that into the event to populate the 
connect arguments with the correct credentials.

On Thu, Sep 7, 2023, at 1:54 PM, Steven Schierholz wrote:
> So I have seen some chats here about cred refresh from vault and some 
> suggestions have been to use @event.listens_for(engine, "do_connect") to 
> update creds when the connection is established. My understanding of this is 
> that connecting to the database should only happen once when my flask 
> application starts up, but I need to update the creds without restarting my 
> application so I'm not sure that the event listener will work in my case.
> 
> Am I understanding that correctly? If so, is there a way to get the right 
> creds to pass to the engine for sqlalchemy every 24 hours when the creds from 
> vault get updated without restarting my application?
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/2de2e3fb-526c-49f7-8780-9ef55a9ad8bcn%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/a6b972d3-0700-4772-a2c8-243706e4c6e9%40app.fastmail.com.

Re: [sqlalchemy] Duplicating a relationship from one ORM model to a subquery model

[Mike Bayer]

send viewonly=True to these relationships you make.  these are not for writing 
anyway and that will resolve the overlaps warnings (the warning says as much).

On Thu, Sep 7, 2023, at 10:41 AM, zedr...@gmail.com wrote:
> Hi Mike,
> 
> Thanks a lot for taking the time to reply…
> 
> Indeed, I came to a similar conclusion and worked out what seems like a 
> fairly clean way to copy relationships (assuming they do not use 
> secondary_join).
> 
> The fact that I cannot seem to update the mapper before the class is created 
> makes my code a bit more convoluted than I'd like… But this seems to work:
> 
> def copy_relationship(source_model, relationship_name):
> source_relationship = getattr(source_model.__mapper__.relationships, 
> relationship_name, None)
> kwargs = {}
> overlaps = [relationship_name]
> for param_name in inspect.signature(relationship).parameters:
> if value := getattr(source_relationship, param_name, None):
> if param_name in ('backref', 'back_populates'):
> overlaps.append(value)
> elif param_name not in ('init', 'primaryjoin'):
> kwargs[param_name] = value
> return relationship(**kwargs, overlaps = ','.join(overlaps))
> 
> class FooViewMeta(type(db.Model)):
> def __new__(mcs, name, bases, attrs):
> attrs["__table__"] = (select(Foo).with_only_columns(Foo.id).subquery())
> attrs["bazs"] = copy_relationship(Foo, 'bazs')
> return super().__new__(mcs, name, (db.Model,), attrs)
> 
> def __init__(cls, name, bases, attrs):
> super().__init__(name, bases, attrs)
> for rel_name, rel_attr in cls.__mapper__.relationships.items():
> expr = getattr(rel_attr, 'primaryjoin', None)
> if not expr: continue
> kwargs = {'operator': expr.operator}
> for attr in ('left', 'right'):
> side = getattr(expr, attr)
> if ('__table__' in side) and (side.table == Foo.__table__) and (side.name in 
> cls.__table__.c):
> side = foreign(getattr(cls.__table__.c, side.name))
> kwargs[attr] = side
> rel_attr.primary_join = BinaryExpression(**kwargs)
> 
> class FooView(metaclass=FooViewMeta):
> pass
> 
> 
> 
> Running the code above without providing an overlaps argument to the new 
> relationship, results in a relationship conflict warning:
> :10: SAWarning: relationship 'FooView.bazs' will copy column 
> anon_1.id to column baz.foo_id, 
> which conflicts with relationship(s): 'Baz.foo' (copies foo.id to 
> baz.foo_id). If this is not the intention,
> consider if these relationships should be linked with back_populates, or if 
> viewonly=True should be applied 
> to one or more if they are read-only. For the less common case that foreign 
> key constraints are partially 
> overlapping, the orm.foreign() annotation can be used to isolate the columns 
> that should be written towards.
> To silence this warning, add the parameter 'overlaps="foo"' to the 
> 'FooView.bazs' relationship. 
> (Background on this warning at: https://sqlalche.me/e/20/qzyx)
> 
> As you can see in the code above, adding foreign to the join expression, does 
> not solve the problem. 
> 
> All in all, I wish there was a more straightforward way to create models that 
> inherit from an existing model, while loading/accessing only a subset of 
> columns (basically, polymorphism, without a polymorphism key)… Will gladly 
> take any suggestion on a different approach…
> 
> But at least this seems to work!
> 
> Thanks again,
> -- 
> Dave
> 
> 
> On Thursday, 7 September 2023 at 13:33:05 UTC+1 Mike Bayer wrote:
>> 
>> 
>> On Thu, Sep 7, 2023, at 4:39 AM, zedr...@gmail.com wrote:
>>> 
>>> *Is there a clean way to (programmatically) duplicate all relationship from 
>>> an existing model, over to a new model (that targets the same table and 
>>> selects a subset of columns as a subquery)?*
>> 
>> relatonships are fixed to their parent class and have internal state that is 
>> dependent on it, so it can't be shared between two unrelated classes.
>> 
>> you would need to introspect the attributes of each relationship that are 
>> important to the application, such as the target class etc., and create a 
>> new relationship() that has those attributes.Assuming the relationships 
>> dont have special join conditions, this would involve just looking for 
>> existing_class.__mapper__.attrs.relationship.entity, which is the mapper to 
>> which it refers, and placing this into a new relationship().
>> 
>> 
>>> 
>> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See 

Re: [sqlalchemy] Duplicating a relationship from one ORM model to a subquery model

[Mike Bayer]

On Thu, Sep 7, 2023, at 4:39 AM, zedr...@gmail.com wrote:
> 
> *Is there a clean way to (programmatically) duplicate all relationship from 
> an existing model, over to a new model (that targets the same table and 
> selects a subset of columns as a subquery)?*

relatonships are fixed to their parent class and have internal state that is 
dependent on it, so it can't be shared between two unrelated classes.

you would need to introspect the attributes of each relationship that are 
important to the application, such as the target class etc., and create a new 
relationship() that has those attributes.Assuming the relationships dont 
have special join conditions, this would involve just looking for 
existing_class.__mapper__.attrs.relationship.entity, which is the mapper to 
which it refers, and placing this into a new relationship().


> 

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/3c23e775-f2d1-487d-8f5d-e7785399aeba%40app.fastmail.com.

Re: [sqlalchemy] sqlalchemy.orm.exc.FlushError on subclass

[Mike Bayer]

if you can't correct this model to apply the persistence details to the 
concrete class you wish to persist and query, then you'd do the suggested 
"enable_typechecks=False".  There is no attribute in SQLAlchemy named "meta" 
and no stack trace is given here so I dont know to what that refers.

Overall I'm not sure how this API_Person class is useful because you can't 
query for them. I would think that if you cant change the original model 
then you'd have this API_Person as a series of helper functions that accept a 
Person as their argument.



On Wed, Sep 6, 2023, at 1:56 PM, 'Luna Lucadou' via sqlalchemy wrote:
> The project I am working on is split up into several modules. Previously, 
> each module had its own ORM classes.
> However, due to several bugs arising from forgetting to update each module's 
> ORM classes in lock step when adding new functionality, we have decided it 
> would be best to extract the ORM classes which interact with our DB into 
> their own module and make that available to the other modules via pip.
> 
> One of these modules, which monitors for changes in an upstream DB and 
> applies them to ours, has some methods which are not present in the other 
> modules and which cannot be easily extracted out due to its dependencies on 
> upstream DB functionality.
> 
> As such, in this module, we must subclass the ORM models which interact with 
> our DB:
> 
> models.apimodels.db.person.py:
> #...
> @dataclass(init=False, eq=True, unsafe_hash=True)
> class Person(Base):
> __tablename__ = "person"
> 
> id: Mapped[int] = mapped_column(primary_key=True)
> first_name: Mapped[str]
> last_name: Mapped[str]
> email_address: Mapped[str]
> office_address: Mapped[str]
> office_phone_number: Mapped[str]
> 
> # ...
> 
> etl.models.api_db.api_person.py:
> #...
> from apimodels.db.person import Person as PersonBase
> # ...
> class API_Person(PersonBase):
> __tablename__ = "person"
> __table_args__ = {"keep_existing": True}
> 
> def get_pvi(self):
> # ...
> 
> def get_historical_pvis(self) -> list[str]:
> # ...
> 
> def __eq__(self):
> # ...
> 
> def __hash__(self):
> # ...
> 
> @staticmethod
> def from_upstream_hub_person(
> uh_person: Optional[UH_Person],
> ) -> Optional["API_Person"]:
> # ...
> 
> Of note is that this subclass does not add any new attributes or modify 
> existing ones, it merely adds some helper methods related to identifying 
> primary key changes in the upstream DB. (This is also why we override the eq 
> and hash methods provided by dataclasses - incoming changesets have to be 
> matched against existing records, even when primary keys change upstream.)
> 
> This is effectively single-table inheritance, but it is not a good fit for 
> polymorphic_identity since it is not a distinct class, merely adding 
> module-specific helper methods, and if I am reading the documentation 
> correctly, using polymorphic_identity would mean any records touched by the 
> API_Person subclass (which is all of them) would no longer be usable by other 
> modules, which do not extend any of the models.
> 
> From what I have read, it seems like the keep_existing param should be of use 
> here, but neither it nor extend_existing set to True help with the errors I 
> am seeing when attempting to interact with this subclass in any way:
> 
> sqlalchemy.orm.exc.FlushError: Attempting to flush an item of type  'model.api_db.api_person.API_Person'> as a member of collection 
> "Identifier.person". Expected an object of type  'apimodels.db.person.Person'> or a polymorphic subclass of this type. If 
>  is a subclass of  'apimodels.db.person.Person'>, configure mapper "Mapper[Person(person)]" to 
> load this subtype polymorphically, or set enable_typechecks=False to allow 
> any subtype to be accepted for flush.
> 
> I did try setting enable_typechecks to False, but this results in a different 
> error when attempting to use getattr on the subclass:
> 
> AttributeError: 'Person' object has no attribute 'meta'
> 
> Is there a better way of doing this?
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/bff6d34f-ea35-4aba-ba94-5ae1f29154fan%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and 

Re: [sqlalchemy] sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'PPESSOA' expected to update 1 row(s); 0 were matched.

[Mike Bayer]

it can be related to the driver or the "cloud" database you're using, we have 
lots of reports of SQL Server drivers not reporting on matched rows correctly

On Wed, Sep 6, 2023, at 7:40 AM, Leandro Lázaro wrote:
> I updated the driver to the latest version and removed the line 
> 'self.engine.dialect.supports_sane_rowcount = False,' and the same error 
> occurred again. Could this driver also have an unknown bug? 
> 
> TCLOUD_DATABASE_DRIVER_V18 = 
> "/opt/microsoft/msodbcsql18/lib64/libmsodbcsql-18.3.so.1.1"
> 
> SQLALCHEMY_TCLOUD_DATABASE_URI = 
> 'mssql://'+TCLOUD_DATABASE_USERNAME+':'+TCLOUD_DATABASE_PASSWORD_UPDATED+'@'+TCLOUD_DATABASE_HOST+':'+TCLOUD_DATABASE_PORT+'/'+TCLOUD_DATABASE_NAME+'?driver='+TCLOUD_DATABASE_DRIVER_V18+'=yes=yes'
> 
> Em ter., 5 de set. de 2023 às 22:03, Leandro Lázaro 
>  escreveu:
>> It worked!!! Thank you very much, it solved my problem. I was already going 
>> crazy haha. I will try to update the driver to solve the problem right at 
>> the root. Just a doubt: Changing to false does not generate any side effects?
>> 
>> Em ter., 5 de set. de 2023 às 21:16, Mike Bayer 
>>  escreveu:
>>> __
>>> we will work around what is likely some kind of driver related error:
>>> 
>>> engine = create_engine(" your engine string normally  .. ")
>>> engine.dialect.supports_sane_rowcount = False
>>> 
>>> 
>>> then run the program normally.  the UPDATE will succeed.  however, after 
>>> the program runs, look in the database and make sure the UPDATE took effect 
>>> correctly.if so, the driver, either pyodbc, or the ODBC driver in use, 
>>> has an unknown bug.   try updating drivers, both pyodbc and the ODBC driver 
>>> in use. 
>>> 
>>> 
>>> On Tue, Sep 5, 2023, at 6:50 PM, Leandro Lázaro wrote:
>>>> Hello, apologies for the delay.
>>>> 
>>>> I'm using SQL Server. Updates using the code below work normally. However, 
>>>> when I try to change the object directly and apply commit, I receive the 
>>>> mentioned error.
>>>> 
>>>> 
>>>> stmt = update(PPESSOA).where(PPESSOA.CODIGO == 
>>>> ppessoa.CODIGO).values(EMAILPESSOAL=data['EMAIL'])
>>>> session.execute(stmt)
>>>> session.commit() 
>>>> 
>>>> The connection string is:
>>>> SQLALCHEMY_TCLOUD_DATABASE_URI = 
>>>> 'mssql://'+TCLOUD_DATABASE_USERNAME+':'+TCLOUD_DATABASE_PASSWORD_UPDATED+'@'+TCLOUD_DATABASE_HOST+':'+TCLOUD_DATABASE_PORT+'/'+TCLOUD_DATABASE_NAME+'?driver='+TCLOUD_DATABASE_DRIVER
>>>> 
>>>> Driver is:
>>>> 
>>>> libmsodbcsql-17.10.so.4.1
>>>> 
>>>> Thank you for all!
>>>> 
>>>> 
>>>> 
>>>> Em terça-feira, 22 de agosto de 2023 às 15:51:47 UTC-3, Mike Bayer 
>>>> escreveu:
>>>>> __
>>>>> if it's fully reproducible every time with both statements, then this 
>>>>> suggests something is happening with the database server itself, such as 
>>>>> some kind of issue with triggers getting involved or something.   In 
>>>>> particular if this is MS SQL Server and there are triggers involved, the 
>>>>> updated rowcount might not be working correctly and extra steps might 
>>>>> need to be taken to turn off SQLAlchemy's rowcount facilities.
>>>>> 
>>>>> you want to turn on `echo=True` and see that the expected UPDATE 
>>>>> statement takes place.  Then you might want to try running those UPDATE 
>>>>> statements directly on the database and see that they work.
>>>>> 
>>>>> What database backend, what database driver, and is the issue only 
>>>>> reproducible in a single environment or can it be reproduced elsewhere?
>>>>> 
>>>>> 
>>>>> On Tue, Aug 22, 2023, at 2:42 PM, Leandro Lázaro wrote:
>>>>>> Hello
>>>>>> 
>>>>>> First, thanks for building this library. God bless you
>>>>>> 
>>>>>> 
>>>>>> I've been trying to understand what's going on for days.
>>>>>> 
>>>>>> This code works fine:
>>>>>> 
>>>>>> https://pastebin.com/fLTnB8jy
>>>>>> 
>>>>>> But if I try to modify any other parameter like PERSONAL EMAIL I get the 
>>>>>> error:
>>>>>> 
>>>>>> sqlalchemy.orm.ex

Re: [sqlalchemy] sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'PPESSOA' expected to update 1 row(s); 0 were matched.

[Mike Bayer]

changing it to false means if an UPDATE statement that expects to update on 
primary key in the ORM does not actually match any rows, the program continues 
and this may be a silent failure, leading to more confusing problems later if 
the row was in fact deleted or is otherwise non-existent.

On Tue, Sep 5, 2023, at 9:03 PM, Leandro Lázaro wrote:
> It worked!!! Thank you very much, it solved my problem. I was already going 
> crazy haha. I will try to update the driver to solve the problem right at the 
> root. Just a doubt: Changing to false does not generate any side effects?
> 
> Em ter., 5 de set. de 2023 às 21:16, Mike Bayer 
>  escreveu:
>> __
>> we will work around what is likely some kind of driver related error:
>> 
>> engine = create_engine(" your engine string normally  .. ")
>> engine.dialect.supports_sane_rowcount = False
>> 
>> 
>> then run the program normally.  the UPDATE will succeed.  however, after the 
>> program runs, look in the database and make sure the UPDATE took effect 
>> correctly.if so, the driver, either pyodbc, or the ODBC driver in use, 
>> has an unknown bug.   try updating drivers, both pyodbc and the ODBC driver 
>> in use. 
>> 
>> 
>> On Tue, Sep 5, 2023, at 6:50 PM, Leandro Lázaro wrote:
>>> Hello, apologies for the delay.
>>> 
>>> I'm using SQL Server. Updates using the code below work normally. However, 
>>> when I try to change the object directly and apply commit, I receive the 
>>> mentioned error.
>>> 
>>> 
>>> stmt = update(PPESSOA).where(PPESSOA.CODIGO == 
>>> ppessoa.CODIGO).values(EMAILPESSOAL=data['EMAIL'])
>>> session.execute(stmt)
>>> session.commit() 
>>> 
>>> The connection string is:
>>> SQLALCHEMY_TCLOUD_DATABASE_URI = 
>>> 'mssql://'+TCLOUD_DATABASE_USERNAME+':'+TCLOUD_DATABASE_PASSWORD_UPDATED+'@'+TCLOUD_DATABASE_HOST+':'+TCLOUD_DATABASE_PORT+'/'+TCLOUD_DATABASE_NAME+'?driver='+TCLOUD_DATABASE_DRIVER
>>> 
>>> Driver is:
>>> 
>>> libmsodbcsql-17.10.so.4.1
>>> 
>>> Thank you for all!
>>> 
>>> 
>>> 
>>> Em terça-feira, 22 de agosto de 2023 às 15:51:47 UTC-3, Mike Bayer escreveu:
>>>> __
>>>> if it's fully reproducible every time with both statements, then this 
>>>> suggests something is happening with the database server itself, such as 
>>>> some kind of issue with triggers getting involved or something.   In 
>>>> particular if this is MS SQL Server and there are triggers involved, the 
>>>> updated rowcount might not be working correctly and extra steps might need 
>>>> to be taken to turn off SQLAlchemy's rowcount facilities.
>>>> 
>>>> you want to turn on `echo=True` and see that the expected UPDATE statement 
>>>> takes place.  Then you might want to try running those UPDATE statements 
>>>> directly on the database and see that they work.
>>>> 
>>>> What database backend, what database driver, and is the issue only 
>>>> reproducible in a single environment or can it be reproduced elsewhere?
>>>> 
>>>> 
>>>> On Tue, Aug 22, 2023, at 2:42 PM, Leandro Lázaro wrote:
>>>>> Hello
>>>>> 
>>>>> First, thanks for building this library. God bless you
>>>>> 
>>>>> 
>>>>> I've been trying to understand what's going on for days.
>>>>> 
>>>>> This code works fine:
>>>>> 
>>>>> https://pastebin.com/fLTnB8jy
>>>>> 
>>>>> But if I try to modify any other parameter like PERSONAL EMAIL I get the 
>>>>> error:
>>>>> 
>>>>> sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'PPESSOA' 
>>>>> expected to update 1 row(s); 0 were matched
>>>>> 
>>>>> Example: https://pastebin.com/EwdwLRcp
>>>>> 
>>>>> Tank you
>>>>> 
>>>>> 
>>>>> --
>>>>> SQLAlchemy -
>>>>> The Python SQL Toolkit and Object Relational Mapper
>>>>>  
>>>>> http://www.sqlalchemy.org/
>>>>>  
>>>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>>>> description.
>>>>> ---
>>>>> You received this message because you are subscribed to the Google Groups 
>>>

Re: [sqlalchemy] sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'PPESSOA' expected to update 1 row(s); 0 were matched.

[Mike Bayer]

we will work around what is likely some kind of driver related error:

engine = create_engine(" your engine string normally  .. ")
engine.dialect.supports_sane_rowcount = False


then run the program normally.  the UPDATE will succeed.  however, after the 
program runs, look in the database and make sure the UPDATE took effect 
correctly.if so, the driver, either pyodbc, or the ODBC driver in use, has 
an unknown bug.   try updating drivers, both pyodbc and the ODBC driver in use. 
 


On Tue, Sep 5, 2023, at 6:50 PM, Leandro Lázaro wrote:
> Hello, apologies for the delay.
> 
> I'm using SQL Server. Updates using the code below work normally. However, 
> when I try to change the object directly and apply commit, I receive the 
> mentioned error.
> 
> 
> stmt = update(PPESSOA).where(PPESSOA.CODIGO == 
> ppessoa.CODIGO).values(EMAILPESSOAL=data['EMAIL'])
> session.execute(stmt)
> session.commit() 
> 
> The connection string is:
> SQLALCHEMY_TCLOUD_DATABASE_URI = 
> 'mssql://'+TCLOUD_DATABASE_USERNAME+':'+TCLOUD_DATABASE_PASSWORD_UPDATED+'@'+TCLOUD_DATABASE_HOST+':'+TCLOUD_DATABASE_PORT+'/'+TCLOUD_DATABASE_NAME+'?driver='+TCLOUD_DATABASE_DRIVER
> 
> Driver is:
> 
> libmsodbcsql-17.10.so.4.1
> 
> Thank you for all!
> 
> 
> 
> Em terça-feira, 22 de agosto de 2023 às 15:51:47 UTC-3, Mike Bayer escreveu:
>> __
>> if it's fully reproducible every time with both statements, then this 
>> suggests something is happening with the database server itself, such as 
>> some kind of issue with triggers getting involved or something.   In 
>> particular if this is MS SQL Server and there are triggers involved, the 
>> updated rowcount might not be working correctly and extra steps might need 
>> to be taken to turn off SQLAlchemy's rowcount facilities.
>> 
>> you want to turn on `echo=True` and see that the expected UPDATE statement 
>> takes place.  Then you might want to try running those UPDATE statements 
>> directly on the database and see that they work.
>> 
>> What database backend, what database driver, and is the issue only 
>> reproducible in a single environment or can it be reproduced elsewhere?
>> 
>> 
>> On Tue, Aug 22, 2023, at 2:42 PM, Leandro Lázaro wrote:
>>> Hello
>>> 
>>> First, thanks for building this library. God bless you
>>> 
>>> 
>>> I've been trying to understand what's going on for days.
>>> 
>>> This code works fine:
>>> 
>>> https://pastebin.com/fLTnB8jy
>>> 
>>> But if I try to modify any other parameter like PERSONAL EMAIL I get the 
>>> error:
>>> 
>>> sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'PPESSOA' 
>>> expected to update 1 row(s); 0 were matched
>>> 
>>> Example: https://pastebin.com/EwdwLRcp
>>> 
>>> Tank you
>>> 
>>> 
>>> --
>>> SQLAlchemy -
>>> The Python SQL Toolkit and Object Relational Mapper
>>>  
>>> http://www.sqlalchemy.org/
>>>  
>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>> description.
>>> ---
>>> You received this message because you are subscribed to the Google Groups 
>>> "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to sqlalchemy+...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/sqlalchemy/c2939c34-4d95-4fe8-b697-afc705b47a11n%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/sqlalchemy/c2939c34-4d95-4fe8-b697-afc705b47a11n%40googlegroups.com?utm_medium=email_source=footer>.
>> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/65034fdf-0743-4f6a-8f4b-7b947364880dn%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/sqlalchemy/65034fdf-0743-4f6a-8f4b-7b947364880dn%40googlegroups.com?utm_medium=email_source=footer>.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/77ef55c9-9c91-48c4-a1df-5e7fe121e3cd%40app.fastmail.com.

Re: [sqlalchemy] StaleDataError on UPDATE for a DATETIME field on multiple consecutive requests

[Mike Bayer]

if the two values are the same, then it will report zero rows matched.   re: 
"very small difference" the MySQL field is likely not storing the microseconds 
portion of the timestamp so if you run several operations in succession the 
timestamps are very likely identical.

On Sat, Sep 2, 2023, at 2:40 PM, Dumitru Gîra wrote:
> Okay, I guess I figured out why this happens. We are using the mysql+mysqldb 
> dialect and we were explicitly setting** 'client_flag': 0** which by default 
> is set to 2 in SQLAlchemy, thus disabling CLIENT_FLAGS.FOUND_ROWS. Removing 
> *'client_flag': 0 *or setting it to *'client_flag': 2 *makes everything to 
> work as expected.
> What I don't understand, is why the disabled FOUND_ROWS causes StaleDatError 
> when the row was actually updated in the database? Could it be because of a 
> very small difference between the previous datetime/timestamp and the update 
> one?
> 
> On Friday, September 1, 2023 at 10:23:56 PM UTC+3 Mike Bayer wrote:
>> __
>> unless the row is being deleted, this would possibly be a driver bug.  the 
>> pymysql driver should work pretty well whereas mysql-connector I would stay 
>> away from.I would need more detail to reproduce beyond that, the SQL 
>> statement has nohting wrong with it.
>> 
>> On Fri, Sep 1, 2023, at 12:09 PM, Dumitru Gîra wrote:
>>> Hi Mike,
>>> thanks for the quick response.
>>> 
>>> These would be the debug logs for
>>> 
>>> import requests
>>> for i in range(2):
>>> r = requests.put('http://0.0.0.0:8080/update-datetime')
>>> print(r.text, r.code)
>>> 
>>> 2023-09-01 16:06:20,711 INFO sqlalchemy.engine.Engine SELECT DATABASE()
>>> 2023-09-01 16:06:20,711 INFO sqlalchemy.engine.Engine [raw sql] ()
>>> 2023-09-01 16:06:20,713 DEBUG sqlalchemy.engine.Engine Col ('DATABASE()',)
>>> 2023-09-01 16:06:20,713 DEBUG sqlalchemy.engine.Engine Row ('dbname',)
>>> 2023-09-01 16:06:20,714 INFO sqlalchemy.engine.Engine SELECT @@sql_mode
>>> 2023-09-01 16:06:20,714 INFO sqlalchemy.engine.Engine [raw sql] ()
>>> 2023-09-01 16:06:20,715 DEBUG sqlalchemy.engine.Engine Col ('@@sql_mode',)
>>> 2023-09-01 16:06:20,715 DEBUG sqlalchemy.engine.Engine Row 
>>> ('ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION',)
>>> 2023-09-01 16:06:20,716 INFO sqlalchemy.engine.Engine SELECT 
>>> @@lower_case_table_names
>>> 2023-09-01 16:06:20,716 INFO sqlalchemy.engine.Engine [raw sql] ()
>>> 2023-09-01 16:06:20,717 DEBUG sqlalchemy.engine.Engine Col 
>>> ('@@lower_case_table_names',)
>>> 2023-09-01 16:06:20,717 DEBUG sqlalchemy.engine.Engine Row (0,)
>>> 2023-09-01 16:06:20,717 INFO sqlalchemy.engine.Engine BEGIN (implicit)
>>> 2023-09-01 16:06:20,719 INFO sqlalchemy.engine.Engine SELECT foo.id AS 
>>> foo_id, foo.last_login AS foo_last_login, foo.name AS foo_name
>>> FROM foo
>>> WHERE foo.id = %s
>>>  LIMIT %s
>>> 2023-09-01 16:06:20,719 INFO sqlalchemy.engine.Engine [generated in 
>>> 0.00016s] (1, 1)
>>> 2023-09-01 16:06:20,720 DEBUG sqlalchemy.engine.Engine Col ('foo_id', 
>>> 'foo_last_login', 'foo_name')
>>> 2023-09-01 16:06:20,720 DEBUG sqlalchemy.engine.Engine Row (1, 
>>> datetime.datetime(2023, 9, 1, 16, 6, 12), '2023-09-01 15:50:46.029445')
>>> 2023-09-01 16:06:20,722 INFO sqlalchemy.engine.Engine UPDATE foo SET 
>>> last_login=%s WHERE foo.id = %s
>>> 2023-09-01 16:06:20,722 INFO sqlalchemy.engine.Engine [generated in 
>>> 0.00014s] (datetime.datetime(2023, 9, 1, 16, 6, 20, 721146), 1)
>>> 2023-09-01 16:06:20,723 INFO sqlalchemy.engine.Engine COMMIT
>>> 192.168.0.1 - - [01/Sep/2023 16:06:20] "PUT /update-datetime HTTP/1.1" 200 -
>>> 2023-09-01 16:06:20,733 INFO sqlalchemy.engine.Engine BEGIN (implicit)
>>> 2023-09-01 16:06:20,733 INFO sqlalchemy.engine.Engine SELECT foo.id AS 
>>> foo_id, foo.last_login AS foo_last_login, foo.name AS foo_name
>>> FROM foo
>>> WHERE foo.id = %s
>>>  LIMIT %s
>>> 2023-09-01 16:06:20,733 INFO sqlalchemy.engine.Engine [cached since 
>>> 0.01402s ago] (1, 1)
>>> 2023-09-01 16:06:20,734 DEBUG sqlalchemy.engine.Engine Col ('foo_id', 
>>> 'foo_last_login', 'foo_name')
>>> 2023-09-01 16:06:20,735 DEBUG sqlalchemy.engine.Engine Row (1, 
>>> datetime.datetime(2023, 9, 1, 16, 6, 21), '2023-09-01 15:50:46.029445')
>>> 2023-09-01 16:06:20,736 INFO sqlalchemy.engine.Engine UPDATE foo SET 
>>> last_login=%s WHERE foo.id = %s
>>> 2023-09-01 16:06:20,736

Re: [sqlalchemy] StaleDataError on UPDATE for a DATETIME field on multiple consecutive requests

[Mike Bayer]

, in commit
> self._transaction.commit(_to_root=self.future)
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", 
> line 832, in commit
> self._prepare_impl()
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", 
> line 811, in _prepare_impl
> self.session.flush()
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", 
> line 3449, in flush
> self._flush(objects)
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", 
> line 3589, in _flush
> transaction.rollback(_capture_exception=True)
>   File 
> "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 
> 70, in __exit__
> compat.raise_(
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", 
> line 211, in raise_
> raise exception
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", 
> line 3549, in _flush
> flush_context.execute()
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", 
> line 456, in execute
> rec.execute(self)
>   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", 
> line 630, in execute
> util.preloaded.orm_persistence.save_obj(
>   File 
> "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 
> 237, in save_obj
> _emit_update_statements(
>   File 
> "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 
> 1035, in _emit_update_statements
> raise orm_exc.StaleDataError(
> sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'foo' expected 
> to update 1 row(s); 0 were matched.
> On Friday, September 1, 2023 at 7:02:56 PM UTC+3 Mike Bayer wrote:
>> __
>> a stale data error with date/time fields is usually a mismatch between the 
>> date/time value you have locally and the one that's in the database,. 
>> assuming you are using the datetime column as a "version_id" in your 
>> mapping. Issues like strings vs. datetimes and/or microseconds portions 
>> being present or non-present are the most common.
>> 
>> would need to see SQL debug logging to have more insight / confirm that's 
>> the problem.
>> 
>> On Fri, Sep 1, 2023, at 11:59 AM, Dumitru Gîra wrote:
>>> I have encountered a very strange behaviour and I don't know where to look 
>>> further.
>>> I have a small API https://pastebin.com/qW01jNz8 with 2 endpoints: one to 
>>> update the name (VARCHAR) and one to update the last_login (DATETIME).
>>> 
>>> If I make 10 consecutive requests to /update-string - everything works as 
>>> expected.
>>> If I make 10 consecutive requests to /update-datetime - the first request 
>>> works and other 9 fail with sqlalchemy.orm.exc.StaleDataError.
>>> NOTE: putting a time.sleep(1) after every request does not raise the error
>>> 
>>> Consecutive requests are being made using the simple snippet
>>> import requests
>>> for i in range(10):
>>> r = requests.put('http://0.0.0.0:8080/update-datetime')
>>> print(r.text, r.code)
>>> 
>>> The source code and the logs are in above link and for the reference I am 
>>> using:
>>> - MySQL 8.0, but the same happens with 5.7
>>> - mysqldb connector, but the same happens with pymysql
>>> - SQLAlchemy==1.4.49
>>> 
>>> Is this related to MySQL and specifically to date/time related fields (as 
>>> the same happens with TIMESTAMP) or can it be something from SQLAlchemy 
>>> side?
>>> Any directions would be highly appreciated.
>>> 
>>> Kind regards,
>>> Dumitru
>>> 
>>> 
>>> --
>>> SQLAlchemy -
>>> The Python SQL Toolkit and Object Relational Mapper
>>>  
>>> http://www.sqlalchemy.org/
>>>  
>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>> description.
>>> ---
>>> You received this message because you are subscribed to the Google Groups 
>>> "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to sqlalchemy+...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/sqlalchemy/e53169bd-617f-41f3-b9d3-8002ea75496an%40googlegroups.com
>>>  
>>> &

Re: [sqlalchemy] StaleDataError on UPDATE for a DATETIME field on multiple consecutive requests

[Mike Bayer]

a stale data error with date/time fields is usually a mismatch between the 
date/time value you have locally and the one that's in the database,. assuming 
you are using the datetime column as a "version_id" in your mapping. Issues 
like strings vs. datetimes and/or microseconds portions being present or 
non-present are the most common.

would need to see SQL debug logging to have more insight / confirm that's the 
problem.

On Fri, Sep 1, 2023, at 11:59 AM, Dumitru Gîra wrote:
> I have encountered a very strange behaviour and I don't know where to look 
> further.
> I have a small API https://pastebin.com/qW01jNz8 with 2 endpoints: one to 
> update the name (VARCHAR) and one to update the last_login (DATETIME).
> 
> If I make 10 consecutive requests to /update-string - everything works as 
> expected.
> If I make 10 consecutive requests to /update-datetime - the first request 
> works and other 9 fail with sqlalchemy.orm.exc.StaleDataError.
> NOTE: putting a time.sleep(1) after every request does not raise the error
> 
> Consecutive requests are being made using the simple snippet
> import requests
> for i in range(10):
> r = requests.put('http://0.0.0.0:8080/update-datetime')
> print(r.text, r.code)
> 
> The source code and the logs are in above link and for the reference I am 
> using:
> - MySQL 8.0, but the same happens with 5.7
> - mysqldb connector, but the same happens with pymysql
> - SQLAlchemy==1.4.49
> 
> Is this related to MySQL and specifically to date/time related fields (as the 
> same happens with TIMESTAMP) or can it be something from SQLAlchemy side?
> Any directions would be highly appreciated.
> 
> Kind regards,
> Dumitru
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/e53169bd-617f-41f3-b9d3-8002ea75496an%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/8ddfacc0-e23b-4672-b05f-92b2517d830a%40app.fastmail.com.

Re: [sqlalchemy] sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'PPESSOA' expected to update 1 row(s); 0 were matched.

[Mike Bayer]

if it's fully reproducible every time with both statements, then this suggests 
something is happening with the database server itself, such as some kind of 
issue with triggers getting involved or something.   In particular if this is 
MS SQL Server and there are triggers involved, the updated rowcount might not 
be working correctly and extra steps might need to be taken to turn off 
SQLAlchemy's rowcount facilities.

you want to turn on `echo=True` and see that the expected UPDATE statement 
takes place.  Then you might want to try running those UPDATE statements 
directly on the database and see that they work.

What database backend, what database driver, and is the issue only reproducible 
in a single environment or can it be reproduced elsewhere?


On Tue, Aug 22, 2023, at 2:42 PM, Leandro Lázaro wrote:
> Hello
> 
> First, thanks for building this library. God bless you
> 
> 
> I've been trying to understand what's going on for days.
> 
> This code works fine:
> 
> https://pastebin.com/fLTnB8jy
> 
> But if I try to modify any other parameter like PERSONAL EMAIL I get the 
> error:
> 
> sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'PPESSOA' 
> expected to update 1 row(s); 0 were matched 
> 
> Example: https://pastebin.com/EwdwLRcp
> 
> Tank you
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/c2939c34-4d95-4fe8-b697-afc705b47a11n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/aafd2f50-d3c7-4018-90e7-3eddbbd9d6db%40app.fastmail.com.

Re: [sqlalchemy] Queue Pool Error

[Mike Bayer]

SQLAlchemy docs have a whole section on what this error means at:

https://docs.sqlalchemy.org/en/20/errors.html#queuepool-limit-of-size-x-overflow-y-reached-connection-timed-out-timeout-z



On Fri, Aug 18, 2023, at 2:34 AM, Krishnendu Ghosh wrote:
> *queue pool limit of size 5 overflow 10 reached connection timed out timeout 
> 30.00* how to solve in python flask_sqlalchemy for gcp cloud sql ?
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/70dba45c-0952-42bd-b51e-bd575dcd94d5n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/915f48e4-dbcd-4cc0-bf60-1bff64b6874a%40app.fastmail.com.

Re: [sqlalchemy] Issuing Raw SQL and Returning a List of Objects

[Mike Bayer]

the raw SQL to ORM mapping pattern has a lot of limitations but it is 
documented at 
https://docs.sqlalchemy.org/en/20/orm/queryguide/select.html#getting-orm-results-from-textual-statements
 .



On Thu, Aug 17, 2023, at 4:26 PM, Mike Graziano wrote:
> To all,
> 
> I am new to Python and SQLAlchemy.  I was a Java developer who used the 
> MyBatis ORM.  I was using PONY ORM for a while, but was concerned that 
> SQLAlchemy is the gold standard for ORMs and Python, but there is something 
> about MyBatis that I can't seem to find in SQLAlchemy, but maybe I have not 
> googled enough.
> 
> In short, I don't want to use SQLAlchemy's select.  I want to issue raw SQL 
> and have SQLAlchemy's ORM capability populate a collection of objects.  That 
> may not be pythonic, but I'd like to know how to do it with declarative 
> mapping.  Is it possible and, if so, I'd love to see an example.  
> 
> Thx & rgds
> 
> mjg
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/17383085-4a83-49d0-bf0a-1653552b7d59n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/ef34613f-69ab-4110-b756-74325b44a224%40app.fastmail.com.

Re: [sqlalchemy] Dogpile session caching

[Mike Bayer]

I will note, in case you didn't see it, that in SQLAlchemy 1.4 (as well as in 
2.0) the caching mechanics themselves have changed compared to what they were 
in SQLAlchemy 1.3.  This change was first made in the 1.4 series to use the new 
do_orm_execute() event model, whereas previously in 1.3 there was a Query 
subclass called CachingQuery or something like that.   for 1.4 and above you 
would need to be using the do_orm_execute() code.

On Wed, Aug 2, 2023, at 11:43 PM, Mike Bayer wrote:
> I've just updated them to use new style queries.   There were no issues and 
> all the examples work by switching `Session.query(Thing)` for 
> `Session.scalars(select(Thing))`. The examples will be on the site within 
> an hour but you can see the changes made in 
> https://github.com/sqlalchemy/sqlalchemy/commit/ab413c3c7b655d408d925dfba47f54c207b9668b
>  .
> 
> 
> 
> On Wed, Aug 2, 2023, at 2:05 PM, 'Joe Black' via sqlalchemy wrote:
>> Hello,
>> 
>> First of all, excellent project!  It's such a powerful tool when you really 
>> master the internals, and after several years I think I'm nearing that 
>> point.  
>> 
>> My question today is related to the dogpile caching example 
>> <https://docs.sqlalchemy.org/en/20/_modules/examples/dogpile_caching/local_session_caching.html>s.
>>   I migrated to 2.x awhile back but the given examples in the documentation 
>> for 2.x show the Session.query interface being used.  I understand this has 
>> been deprecated starting with 2.x, so naturally I've migrated existing code 
>> using Session.query.  This seems to break the example code for dogpile 
>> caching.
>> 
>> I was hoping to get some advice on how to implement the caching using the 
>> new unified sa.select and Session.execute/Session.scalars interface, or even 
>> whether dogpile cache supports this interface yet?
>> 
>> Thanks in advance,
>> 
>> Joe
>> 
>> 
>> --
>> SQLAlchemy -
>> The Python SQL Toolkit and Object Relational Mapper
>>  
>> http://www.sqlalchemy.org/
>>  
>> To post example code, please provide an MCVE: Minimal, Complete, and 
>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>> description.
>> ---
>> You received this message because you are subscribed to the Google Groups 
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to sqlalchemy+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/sqlalchemy/29559c02-782b-4d71-be75-82494c5f253dn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/sqlalchemy/29559c02-782b-4d71-be75-82494c5f253dn%40googlegroups.com?utm_medium=email_source=footer>.
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/42ae3b77-3bdf-4ee4-a571-086a3a06a8ba%40app.fastmail.com
>  
> <https://groups.google.com/d/msgid/sqlalchemy/42ae3b77-3bdf-4ee4-a571-086a3a06a8ba%40app.fastmail.com?utm_medium=email_source=footer>.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/ba2c7c0a-821b-42db-b85f-f9a7bde9a1f0%40app.fastmail.com.

Re: [sqlalchemy] Dogpile session caching

[Mike Bayer]

I've just updated them to use new style queries.   There were no issues and all 
the examples work by switching `Session.query(Thing)` for 
`Session.scalars(select(Thing))`. The examples will be on the site within 
an hour but you can see the changes made in 
https://github.com/sqlalchemy/sqlalchemy/commit/ab413c3c7b655d408d925dfba47f54c207b9668b
 .



On Wed, Aug 2, 2023, at 2:05 PM, 'Joe Black' via sqlalchemy wrote:
> Hello,
> 
> First of all, excellent project!  It's such a powerful tool when you really 
> master the internals, and after several years I think I'm nearing that point. 
>  
> 
> My question today is related to the dogpile caching example 
> s.
>   I migrated to 2.x awhile back but the given examples in the documentation 
> for 2.x show the Session.query interface being used.  I understand this has 
> been deprecated starting with 2.x, so naturally I've migrated existing code 
> using Session.query.  This seems to break the example code for dogpile 
> caching.
> 
> I was hoping to get some advice on how to implement the caching using the new 
> unified sa.select and Session.execute/Session.scalars interface, or even 
> whether dogpile cache supports this interface yet?
> 
> Thanks in advance,
> 
> Joe
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/29559c02-782b-4d71-be75-82494c5f253dn%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/42ae3b77-3bdf-4ee4-a571-086a3a06a8ba%40app.fastmail.com.

Re: [sqlalchemy] Sqlacodegen freeze when generate models.py

[Mike Bayer]

hi -

sqlacodegen issues should be discussed at 
https://github.com/agronholm/sqlacodegen/discussions/categories/q-a

On Tue, Aug 1, 2023, at 7:57 AM, Leandro Lázaro wrote:
> Hello,
> When I use the command sqlacodegen mssql+pymssql://USER:PASS@IP:PORT/DB > 
> model.py, the process hangs and continues running for several hours without 
> any results. The database I'm trying to generate models.py for is quite large 
> (over 1000 tables). Is the execution time normal, or is something wrong? I've 
> tested the connection string with engine.connect(), and it seems to be 
> working fine. Am I doing something wrong? Thank you in advance.
> 
> Best regards,
> 
> Leandro L. A. Vieira
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/1a9d575a-f87e-4ff8-a6de-dac0fd3b27fdn%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/1cf76d39-7a6f-434a-9a42-4833a7e8c758%40app.fastmail.com.

Re: [sqlalchemy] Seeking experience/suggestions regarding connection pools

[Mike Bayer]

On Thu, Jul 13, 2023, at 8:07 PM, jens.t...@gmail.com wrote:
> Hello,
> 
> The SQLA Pooling section 
>  says that the default 
> connection pool (QueuePool is the default pool, isn’t it?) has 5 connections 
> with an overflow of 10 (docs 
> ).
> 
> However, in our configuration we use
> 
> *sqlalchemy.pool_recycle = 3600
> sqlalchemy.pool_pre_ping = True
> sqlalchemy.pool_size = 20
> sqlalchemy.max_overflow = 30*
> 
> Independently of how many connections the PG server accepts (looks like 100 
> by default, docs 
> ) a 
> single gRPC server process has a global Engine instance with a connection 
> pool of that configured size.
> 
> Without too many requests flowing through our gRPC server process, we already 
> see these errors:
> 
> *QueuePool limit of size 20 overflow 30 reached, connection timed out, 
> timeout 30.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)*
> 
> Assuming that we have the implementation correct (a global Engine and its 
> connection pool, sessions fetching and releasing connections correctly) the 
> error can mean one of two things:
>  1. too many simultaneous requests hold a connection too long; or
>  2. these settings aren’t realistic for a production server environment.


well 1 and 2 are the same thing.  if you are using a webserver that allows 80 
simultaneous requests that are all using the database, that will use up the 50 
connections you have allocated.   or, if you have some kind of long running 
worker threads or something that are using up connections, that will do it too. 
 finally, if your app is *leaking* connections, that is, opening connections 
and not closing them while continuing to check out more connections at the same 
time, that causes this problem as well.  it's kind of hard to do that though 
unless you have some strange programming patterns in use.

When you get that error, Python will give you a stack trace where the 
connection is being checked out. you can also turn on pool logging to watch 
when connections are checked out and returned.   if you want to test some 
request-handling code in isolation and ensure it uses only one connection, 
there's a special pool called the AssertionPool 
(https://docs.sqlalchemy.org/en/20/core/pooling.html#sqlalchemy.pool.AssertionPool)
 which might be helpful.

> Is there another angle I might be missing? What configuration settings would 
> you recommend? Other than turning on debugging (docs 
> )
>  what tips do you have to debug the issue?
> 
> Much thanks!
> Jens
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/0cdbdb8a-0cff-47f4-b0ca-fccd70ded2den%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/b465c695-bc35-4386-9d1a-82da190186f6%40app.fastmail.com.

Re: [sqlalchemy] Usage of sqlalchemy Uuid type in declarative mapping

[Mike Bayer]

On Thu, Jul 6, 2023, at 1:21 PM, Pierre Massé wrote:
> Dear all,
> 
> I am currently working on implementing UUID as primary keys in my database.
> 
> I have come across an issue that I cannot find an explanation for: using the 
> `sqlalchemy.Uuid` type in declarative mapping yields different results at 
> initialization. I sometimes get the following error:
> 
> *"The type provided inside the 'id' attribute Mapped annotation is the 
> SQLAlchemy type . Expected a Python 
> type instead"*

this message is referring to the difference between *Python* types and 
*SQLAlchemy* types.  A SQLAlchemy type is like "sqlalchemy.Uuid" as you are 
using.  The Python type is the Python UUID type: 
https://docs.python.org/3/library/uuid.html#uuid.UUID

the default lookup is documented here: 
https://docs.sqlalchemy.org/en/20/orm/declarative_tables.html#mapped-column-derives-the-datatype-and-nullability-from-the-mapped-annotation

so you would want

import uuid

class XYZ(...):
# ...

   id: Mapped[uuid.UUID] = mapped_column(...)







> 
> I have a Caregiver table, for which id is a PostgreSQL UUID, and a Match 
> Event table holding 2 fields which use the Caregiver id as foreign key. Those 
> 2 fields are PostgreSQL UUID as well.
> 
> If my classes definitions are as follow, the initialization runs smoothly, as 
> well as inserts in the Match Event table:
> 
> from sqlalchemy import (
> ForeignKey,
> Uuid,
> )
> from sqlalchemy.orm import (
> Mapped,
> mapped_column,
> relationship,
> )
> 
> class MatchEventOrm(Base):
> __tablename__ = "match_event"
> 
> id: Mapped[int] = mapped_column(primary_key=True)
> subject_caregiver_id: Mapped[Uuid] = mapped_column(
> ForeignKey("caregiver.id")
> )
> object_caregiver_id: Mapped[Uuid] = mapped_column(
> ForeignKey("caregiver.id")
> )
> subject_caregiver: Mapped["CaregiverOrm"] = relationship(
> back_populates="match_event_subject_of",
> foreign_keys=[subject_caregiver_id],
> )
> object_caregiver: Mapped["CaregiverOrm"] = relationship(
> back_populates="match_event_object_of",
> foreign_keys=[object_caregiver_id],
> )
> 
> class CaregiverOrm(Base):
> __tablename__ = "caregiver"
> 
> id: Mapped[*str*] = mapped_column(primary_key=True)
> uid: Mapped[str]
> user_email: Mapped[str]
> 
> ...
> 
> 
> 
> However, if I change to the following:
> from sqlalchemy import (
> ForeignKey,
> Uuid,
> )
> from sqlalchemy.orm import (
> Mapped,
> mapped_column,
> relationship,
> )
> 
> class MatchEventOrm(Base):
> __tablename__ = "match_event"
> 
> id: Mapped[int] = mapped_column(primary_key=True)
> subject_caregiver_id: Mapped[Uuid] = mapped_column(
> ForeignKey("caregiver.id")
> )
> object_caregiver_id: Mapped[Uuid] = mapped_column(
> ForeignKey("caregiver.id")
> )
> subject_caregiver: Mapped["CaregiverOrm"] = relationship(
> back_populates="match_event_subject_of",
> foreign_keys=[subject_caregiver_id],
> )
> object_caregiver: Mapped["CaregiverOrm"] = relationship(
> back_populates="match_event_object_of",
> foreign_keys=[object_caregiver_id],
> )
> 
> class CaregiverOrm(Base):
> __tablename__ = "caregiver"
> 
> id: Mapped[*Uuid*] = mapped_column(primary_key=True)
> uid: Mapped[str]
> user_email: Mapped[str]
> 
> ...
> 
> 
> I get the aforementioned error message.
> 
> I may be using type annotations wrong, any insight on what I am doing wrong 
> would be welcome.
> 
> Thanks a lot for your support.
> 
> Regards,
> 
> Pierre
> 
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/CAH4TWVuadpW3d5%3Dt3pgs8DU55Wpx%3DVGBajBKQiskzc87f-aiMw%40mail.gmail.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/4b55fc73-d826-4842-a045-f1610bdc8aa6%40app.fastmail.com.

Re: [sqlalchemy] Soft Delete Pattern

[Mike Bayer]

this is a major area of functionality that begins with the 
"with_loader_criteria" feature: 
https://docs.sqlalchemy.org/en/20/orm/queryguide/api.html#sqlalchemy.orm.with_loader_criteria

integration to make the criteria automatic follows at 
https://docs.sqlalchemy.org/en/20/orm/session_events.html#adding-global-where-on-criteria
 

there's an example of this here: 
https://docs.sqlalchemy.org/en/20/_modules/examples/extending_query/filter_public.html


On Mon, Jul 3, 2023, at 11:13 PM, Andrew Martin wrote:
> Hello,
> 
> I have a base class I tend to use that includes a Boolean is_deleted field so 
> that pretty much every object has that available.
> 
> Is there a good pattern for setting a filter on these objects that 
> automatically adds a WHERE is_deleted = 'false'?
> 
> Or does that just have to be added as a filter on every .query(...) statement?
> 
> 
> Thanks,
> 
> andrew
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/08e50347-ebd9-47a4-b9f8-43b95ebe4e6bn%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/7f628110-e543-492d-8acc-82bd577f1477%40app.fastmail.com.

Re: [sqlalchemy] Left join and nested inner join

[Mike Bayer]

use the join construct directly

from sqlalchemy.orm import join


stmt = select(A).outerjoin(join(B, C), A.id == B.a_id)



On Mon, Jul 3, 2023, at 8:29 AM, Michael Ekoka wrote:
> 
> Hi, I'm looking for the SQLAlchemy equivalent to the query
> 
> SELECT *
> FROM a 
> LEFT OUTER JOIN (b INNER JOIN c ON b.id = c.b_id) 
> ON a.id = b.a_id
> 
> Related:
> https://stackoverflow.com/a/56815807/56974
> https://stackoverflow.com/questions/25514160/nested-joins-in-sqlalchemy
> 
> Table "b" and "c" are joined and filtered first, then the outer join is 
> applied. I was able to achieve the same results using a subquery, whose 
> fields I was subsequently able to load using `contains_eager`. FYI
> 
> subq = session.query(B).join(C).subquery(with_labels=True)
> q = (session.query(A)
>  .outerjoin(subq, A.id==subq.c.b_a_id)
>  .options(contains_eager(A.b, alias=subq)
>   .options(contains_eager(B.c, alias=subq
> r = q.all()
> 
> I'm curious whether there's an equivalent using the above nested join syntax.
> 
> Thanks.
> 
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/86b6cc02-be68-400f-9a13-ef486b560329n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/281524be-e25c-4e37-ad18-397a0908d53b%40app.fastmail.com.

Re: [sqlalchemy] orm.column_property()

[Mike Bayer]

On Fri, Jun 23, 2023, at 9:38 AM, Julien Cigar wrote:
> On Fri, Jun 23, 2023 at 09:29:27AM -0400, Mike Bayer wrote:
>> 
>> Hi Julien -
>
> Hi Mike,
>
>> 
>> OK, if we're going to do one of your big queries, can we please start fresh, 
>> make a new discussion at 
>> https://github.com/sqlalchemy/sqlalchemy/discussions , give me **really 
>> really succinct** models + the query you want, etc. 
>> 
>> the google groups thing here is not exactly going away but it's an 
>> antiquated interface at this point.
>> 
>
> I think you missed one message, I added nesting=True to the cte()
> expression and it does what I want :)

hooray!  onto the weekend then


>
> Next time I'll use https://github.com/sqlalchemy/sqlalchemy/discussions
>
> Thanks!
>
> Julien
>
>> 
>> 
>> On Fri, Jun 23, 2023, at 4:48 AM, Julien Cigar wrote:
>> > On Wed, Jun 21, 2023 at 08:35:36AM -0400, Mike Bayer wrote:
>> >> 
>> >> 
>> >> 
>> >> On Wed, Jun 21, 2023, at 5:12 AM, Julien Cigar wrote:
>> >> > Hello,
>> >> >
>> >> > I'm trying to add a column_property to recursively load and merge some
>> >> > json column.
>> >> >
>> >> > I have the following:
>> >> > https://gist.github.com/silenius/1af1072abae5829f54584f1ea554e074 but I
>> >> > cannot figure how can I replace the '5011' at line 11 (which is the
>> >> > object primary key) with the "current" involved objet..?
>> >> >
>> >> > For example when I'm doing dbsession.get(MyClass, 1234) I cannot figure
>> >> > how reference the id in the column_property query
>> >> 
>> >> well you wouldnt hardcode the id like that, you'd need to make your query 
>> >> correlated to the parent.
>> >> 
>> >> so it would select() from "ac" where "ac" is "aliased(class_)" , then 
>> >> correlate to the class
>> >> 
>> >> 
>> >> .where(
>> >> class_.id == ac.id 
>> >> ).
>> >
>> > Thank you Mike, it _almost_ work.
>> > With (1) SQLAlchemy now generates (2). I'm still getting an error:
>> >
>> > sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) 
>> > invalid reference to FROM-clause entry for table "content"
>> > LINE 4: WHERE content_1.id = content.id UNION ALL SELECT content.id ...
>> >  ^
>> > HINT:  Perhaps you meant to reference the table alias "content_1".
>> >
>> > This is because the WITH statement appears at the "top" rather than a
>> > subselect (and cannot reference the outer table)
>> >
>> > Any idea how could I transform (1) to generate (3) rather than (2) ?
>> >
>> > (1) https://gist.github.com/silenius/43717848057d87ef7945c4d621fce5e8
>> > (2) https://gist.github.com/silenius/34d3a319a52f8fe4fcd2b6418ca9e927
>> > (3) https://gist.github.com/silenius/6d048f2b7819d39b70cc4d77237f9031
>> >
>> > Thanks a lot!
>> >
>> > Julien
>> >
>> >> 
>> >> 
>> >> as far as how that integrates into the CTE and all that you'd have to 
>> >> arrange the geometry of the SELECT to work out correctly, I would start 
>> >> with something very simple and build outwards.
>> >> 
>> >> correlated subquery column_property at 
>> >> https://docs.sqlalchemy.org/en/20/orm/mapped_sql_expr.html#using-column-property
>> >> 
>> >> -- 
>> >> SQLAlchemy - 
>> >> The Python SQL Toolkit and Object Relational Mapper
>> >> 
>> >> http://www.sqlalchemy.org/
>> >> 
>> >> To post example code, please provide an MCVE: Minimal, Complete, and 
>> >> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
>> >> description.
>> >> --- 
>> >> You received this message because you are subscribed to the Google Groups 
>> >> "sqlalchemy" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send an 
>> >> email to sqlalchemy+unsubscr...@googlegroups.com.
>> >> To view this discussion on the web visit 
>> >> https://groups.google.com/d/msgid/sqlalchemy/400b6b72-f601-47c8-aa30-827240097eb8%40app.fastmail.com.
>> >
>> > -- 
>> > Julien Cigar
>> > Belgian 

Re: [sqlalchemy] Problems with multiple consecutive joins

[Mike Bayer]

hi -

same as the other message I just answered here, nobody can tell what would be 
wrong with this query without a runnable example, please post **very succinct** 
sample classes + the query you are seeking at 
https://github.com/sqlalchemy/sqlalchemy/discussions

On Fri, Jun 23, 2023, at 6:10 AM, Piotr wrote:
> I have a problems with a CTE query using multiple consecutive joins (A -> B 
> -> C -> D). A and B have both relation to other table so they are joined by 
> that other_id value. C has FK to B and C-D are One to One (id=id).
> 
> The query looks like so:
> 
> latest_message = (session.query(
> models.Main.id.label('main_id'),
> message_models.Third.id,
> message_models.Third.type,
> message_models.Fourth.status,
> ).select_from(
> models.Main, models.Second, message_models.Third, message_models.Fourth,
> ).join(
> models.Second, models.Main.other_id == models.Second.other_id,
> ).join(
> message_models.Third, message_models.Third.second_id == models.Second.id,
> ).join(
> message_models.Fourth, message_models.Fourth.id == 
> message_models.Third.id,
> ).filter(
> models.Main.some_feld == 'abcd',
> ).distinct(
> models.Main.id,
> ).order_by(
>models.Main.id,
>message_models.Third.created_at.desc(),
> ).cte('latest_message'))
> 
> Which results in an error:
> 
> *Can't determine which FROM clause to join from, there are multiple FROMS 
> which can join to this entity. Please use the .select_from() method to 
> establish an explicit left side, as well as providing an explicit ON clause 
> if not present already to help resolve the ambiguity.*
> 
> If I omit explicit join rules it manages to generate an SQL but it's not 
> valid:
> 
> *(psycopg2.errors.DuplicateAlias) table name "third_table" specified more 
> than once*
> **
> And looking at it it does generate weird joins.
> 
> Written by hand in SQL it works as needed - something like this:
> 
> latest_message AS (
> SELECT DISTINCT ON (main_table.id) main_table.id as "main_id", 
> third_table.id,
> third_table.type, fourth_table.status
> FROM main_table
> JOIN second_table ON second_table.other_id = main_table.other_id
> JOIN third_table ON
> third_table.second_id = second_table.id
> JOIN fourth_table ON fourth_table.id = third_table.id
> WHERE main_table.some_feld = 'abcd'
> ORDER BY main_table.id, third_table.created_at DESC
> )
> 
> Anyone know how to fix the SQLAlchemy query version?
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/db95cc7b-9a6e-4225-b58e-2566d1262d25n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/6a61b446-ec56-49b4-bd3e-deb6bbfa5061%40app.fastmail.com.

Re: [sqlalchemy] orm.column_property()

[Mike Bayer]

Hi Julien -

OK, if we're going to do one of your big queries, can we please start fresh, 
make a new discussion at https://github.com/sqlalchemy/sqlalchemy/discussions , 
give me **really really succinct** models + the query you want, etc. 

the google groups thing here is not exactly going away but it's an antiquated 
interface at this point.



On Fri, Jun 23, 2023, at 4:48 AM, Julien Cigar wrote:
> On Wed, Jun 21, 2023 at 08:35:36AM -0400, Mike Bayer wrote:
>> 
>> 
>> 
>> On Wed, Jun 21, 2023, at 5:12 AM, Julien Cigar wrote:
>> > Hello,
>> >
>> > I'm trying to add a column_property to recursively load and merge some
>> > json column.
>> >
>> > I have the following:
>> > https://gist.github.com/silenius/1af1072abae5829f54584f1ea554e074 but I
>> > cannot figure how can I replace the '5011' at line 11 (which is the
>> > object primary key) with the "current" involved objet..?
>> >
>> > For example when I'm doing dbsession.get(MyClass, 1234) I cannot figure
>> > how reference the id in the column_property query
>> 
>> well you wouldnt hardcode the id like that, you'd need to make your query 
>> correlated to the parent.
>> 
>> so it would select() from "ac" where "ac" is "aliased(class_)" , then 
>> correlate to the class
>> 
>> 
>> .where(
>> class_.id == ac.id 
>> ).
>
> Thank you Mike, it _almost_ work.
> With (1) SQLAlchemy now generates (2). I'm still getting an error:
>
> sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) 
> invalid reference to FROM-clause entry for table "content"
> LINE 4: WHERE content_1.id = content.id UNION ALL SELECT content.id ...
>  ^
> HINT:  Perhaps you meant to reference the table alias "content_1".
>
> This is because the WITH statement appears at the "top" rather than a
> subselect (and cannot reference the outer table)
>
> Any idea how could I transform (1) to generate (3) rather than (2) ?
>
> (1) https://gist.github.com/silenius/43717848057d87ef7945c4d621fce5e8
> (2) https://gist.github.com/silenius/34d3a319a52f8fe4fcd2b6418ca9e927
> (3) https://gist.github.com/silenius/6d048f2b7819d39b70cc4d77237f9031
>
> Thanks a lot!
>
> Julien
>
>> 
>> 
>> as far as how that integrates into the CTE and all that you'd have to 
>> arrange the geometry of the SELECT to work out correctly, I would start with 
>> something very simple and build outwards.
>> 
>> correlated subquery column_property at 
>> https://docs.sqlalchemy.org/en/20/orm/mapped_sql_expr.html#using-column-property
>> 
>> -- 
>> SQLAlchemy - 
>> The Python SQL Toolkit and Object Relational Mapper
>> 
>> http://www.sqlalchemy.org/
>> 
>> To post example code, please provide an MCVE: Minimal, Complete, and 
>> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
>> description.
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to sqlalchemy+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/sqlalchemy/400b6b72-f601-47c8-aa30-827240097eb8%40app.fastmail.com.
>
> -- 
> Julien Cigar
> Belgian Biodiversity Platform (http://www.biodiversity.be)
> PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> No trees were killed in the creation of this message.
> However, many electrons were terribly inconvenienced.
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/fdxvwszr6d6k624egxak3ctr5wcu43daz4z6o7okhikglq57to%40puqqqclybeig.
>
> Attachments:
> * signature.asc

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/1a27eba0-7528-4c73-be98-d3cf27c23b10%40app.fastmail.com.

Re: [sqlalchemy] Getting a triple of related id fields

[Mike Bayer]

they are perfectly fine to use, nothing is going away with any of that, so take 
any amount of time to migrate them (or not).


On Wed, Jun 21, 2023, at 1:58 PM, Dan Stromberg wrote:
> On Tue, Jun 20, 2023 at 3:47 PM Mike Bayer
>  wrote:
>>
>>
>> step 1 is stop using that silly Flask extension that gives you 
>> "Pipeline.query", I can't tell what it is you want to SELECT from either by 
>> reading this query.
>
> Wow, that made things a lot easier.
>
> Is there any reason to avoid mixing Model.query with direct SQLAlchemy
> calls?  We have a bunch of Model.query right now.  I'm wondering if
> we'd need to rewrite all those upfront, or if we could redo them as
> time permits.
>
> Thanks!
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/CAOvKW55pTerGhFw6F7%3D%3DCgbz0fu2N71uUOgk73A91jP3M10AEQ%40mail.gmail.com.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/0162ae67-412e-470c-88af-7741febdf319%40app.fastmail.com.

Re: [sqlalchemy] orm.column_property()

[Mike Bayer]

On Wed, Jun 21, 2023, at 5:12 AM, Julien Cigar wrote:
> Hello,
>
> I'm trying to add a column_property to recursively load and merge some
> json column.
>
> I have the following:
> https://gist.github.com/silenius/1af1072abae5829f54584f1ea554e074 but I
> cannot figure how can I replace the '5011' at line 11 (which is the
> object primary key) with the "current" involved objet..?
>
> For example when I'm doing dbsession.get(MyClass, 1234) I cannot figure
> how reference the id in the column_property query

well you wouldnt hardcode the id like that, you'd need to make your query 
correlated to the parent.

so it would select() from "ac" where "ac" is "aliased(class_)" , then correlate 
to the class


.where(
class_.id == ac.id 
).


as far as how that integrates into the CTE and all that you'd have to arrange 
the geometry of the SELECT to work out correctly, I would start with something 
very simple and build outwards.

correlated subquery column_property at 
https://docs.sqlalchemy.org/en/20/orm/mapped_sql_expr.html#using-column-property

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/400b6b72-f601-47c8-aa30-827240097eb8%40app.fastmail.com.

Re: [sqlalchemy] Getting a triple of related id fields

[Mike Bayer]

step 1 is stop using that silly Flask extension that gives you 
"Pipeline.query", I can't tell what it is you want to SELECT from either by 
reading this query.

looks like you have pipeline_alias1 and pipeline_alias2, but you are selecting 
from plain Pipeline, nothing is joining to it however.

what do you actually want to SELECT from in the columns clause?

In any case, use select() with join_from():   
https://docs.sqlalchemy.org/en/20/tutorial/data_select.html#tutorial-select-join
   that way there is no ambiguity.

On Tue, Jun 20, 2023, at 5:51 PM, Dan Stromberg wrote:
> I've been banging on this for hours, but I seem to be getting nowhere.
>
> I've tried more things that I can count, but here are two of my attempts:
>
> #  result = (
> #Pipeline.query
> #  .select_from(Storage, NewProduct)
> #  .join(Storage, pipeline_alias1.storage_id == Storage.id)
> #  .join(NewProduct, Storage.product_id == NewProduct.id)
> #  .filter(pipeline_alias2.storage_id == storage_alias1.id)
> #  .filter(storage_alias2.product_id == product_alias1.id)
> #  )
>
>   result = (
> Pipeline.query
>   .select_from(Pipeline, Storage, NewProduct)
>   .join(Storage, pipeline_alias1.storage_id == storage_alias1.id)
>   .join(NewProduct, storage_alias2.product_id == product_alias1.id)
>   )
>
> I keep getting:
> sqlalchemy.exc.InvalidRequestError: Can't determine which FROM clause
> to join from, there are multiple FROMS which can join to this entity.
> Please use the .select_from() method to establish an explicit left
> side, as well as providing an explicit ON clause if not present
> already to help resolve the ambiguity.
>
> How can I tell SQLAlchemy which FROM to use?
>
> Thanks!
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/CAOvKW57frKP1EACaQ4zoRwu7kDAUULAbXx1xFi8eVdw69tZCZg%40mail.gmail.com.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/e5af2441-6503-48f1-9c2a-e544f40d6841%40app.fastmail.com.

Re: [sqlalchemy] Turning a complex query into a view for SQLAlchemy?

[Mike Bayer]

SQLAlchemy has no intrinsic issue with views but the thing that goes wrong with 
views is that they perform very poorly when used with composition.  that is, if 
you take your view and use in a subquery, do some GROUP BY on it, do some LEFT 
OUTER JOIN onto it, the performance plummets, because the view itself is 
already pre-optimized to the complex query which it represents.

so if you are using SQLAlchemy and then adding lots of joinedload() to a query, 
you would get a lot of LEFT OUTER JOIN onto your view and it might not be very 
efficient from a DB server perspective.   solution, dont use joinedload(), we 
have other eager loaders namely selectinload() which put much less stress on 
the query planner.

On Fri, Jun 16, 2023, at 10:15 AM, Dan Stromberg wrote:
> Hi.
>
> In https://pajhome.org.uk/blog/10_reasons_to_love_sqlalchemy.html it says:
>
> When performing highly complex queries, it is possible to define these
> with SQLAlchemy syntax. However, I find there's a certain level of
> complexity where it becomes easier to write SQL directly. In that
> case, you can define a database view that encompasses the complex
> query, and SQLAlchemy can map the view to Python objects.
>
> I spoke with my team lead about this practice, and he was open to
> trying it but said he had heard that SQLAlchemy may have problems
> using views.
>
> Does anyone on the list have anything to add here?
>
> Thanks!
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/CAOvKW54y462yR4m5pOV5ukACA7v0CgCDUcrDztRBogtuE9t_GA%40mail.gmail.com.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/c79a5a5b-dcd3-4b05-aefb-76790b4021dc%40app.fastmail.com.

Re: [sqlalchemy] Re: Trying to find a way to sort a query given the name of one of its columns

[Mike Bayer]

On Mon, Jun 12, 2023, at 3:12 AM, Lele Gaifax wrote:
> Lele Gaifax  writes:
>
>> I will study the contains_eager() alternative and try to modernize my
>> code that still uses that idiom.
>
> After reading the different sections (narrative and reference), I wonder
> if there is a case where the joinedload() has some advantage over
> contains_eager(), or if the former, being "the oldest style of eager
> loading in SA ORM" is kept mainly for backward compatibility.

joinedload() is a lot easier to use for sure and yes there are probably some 
very nested/ crazyish cases where joinedload "works" and contains_eager either 
doesnt quite work or is very difficult to make it work.


>
> bye, lele.
> -- 
> nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
> real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
> l...@metapensiero.it  | -- Fortunato Depero, 1929.
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/87r0qh9mrs.fsf%40metapensiero.it.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/cc58ebb7-6489-4e05-8d21-18bd68b91933%40app.fastmail.com.

Re: [sqlalchemy] Trying to find a way to sort a query given the name of one of its columns

[Mike Bayer]

it looks like you're trying to add an ORDER BY to the table that's only there 
via joinedload().That's *really* not something we anticipate and it would 
be better if people proposed perhaps ad-hoc order_by expressions to be added to 
common loader options like joinedload() and selectinload(), in the same way 
that we offer ad-hoc WHERE criteria for these options now.   as you are 
probably aware, the current way to do "joinedload with custom criteria / 
ordering / etc" is to write the query using outerjoin() and order_by() 
normally, then use contains_eager().

the general thing we see here looks like when your "col_by_name()" routine 
runs, it's getting a Column object that's associated with a different alias 
than the one that is used when joinedload() writes out the joins.   I'm not 
sure offhand if the issue is in col_by_name() or something architectural that 
makes the joinedload() join receive additional processing that isn't including 
the order_by.It's hard for us to support this kind of transformation for 
joined eager loads.



On Sat, Jun 10, 2023, at 12:34 PM, Lele Gaifax wrote:
> Hi,
>
> I spent some more time to improve support for SA 2 of one of my SA-based
> libraries[1] (the most ancient one, born with SA 0.5, fifteen years ago!):
> its goal is to provide a layer that made it easier to "expose" a SA
> query (either an ORM one or a Core select()) thru a web service,
> handling particular arguments to apply different kind of "refinements"
> such as ordering and filtering.
>
> One of the tests that exercise the "ordering" refinements fails, and I
> could not understand what is going wrong nor find a workaround.
>
> The following simple script builds an ORM query and prints its SQL:
>
> import sqlalchemy as sa
>
>
> SQLALCHEMY_VERSION = tuple(int(p) if p.isdigit() else p for p in 
> sa.__version__.split('.'))
>
>
> metadata = sa.MetaData()
>
>
> if SQLALCHEMY_VERSION > (2, 0):
> from sqlalchemy.orm import DeclarativeBase
>
> class Base(DeclarativeBase):
> metadata = metadata
> else:
> from sqlalchemy.orm import declarative_base
>
> Base = declarative_base(metadata=metadata)
>
>
> class Person(Base):
> __tablename__ = 'persons'
>
> id = sa.Column(sa.Integer, primary_key=True)
> firstname = sa.Column(sa.String)
>
>
> class Pet(Base):
> __tablename__ = 'pets'
>
> id = sa.Column(sa.Integer, primary_key=True)
> name = sa.Column(sa.String)
> person_id = sa.Column(sa.Integer, sa.ForeignKey('persons.id'))
>
> person = sa.orm.relationship(Person, 
> backref=sa.orm.backref('pets', order_by=id))
>
>
> engine = sa.create_engine('sqlite:///:memory:')
> Session = sa.orm.sessionmaker(bind=engine)
>
> metadata.create_all(engine)
>
>
> query = 
> sa.orm.Query([Pet]).options(sa.orm.joinedload(Pet.person).load_only(Person.firstname))
> print(str(query))
>
> It emits the following output under both SA 1.4 and SA 2.0:
>
> SELECT pets.id AS pets_id, pets.name AS pets_name, pets.person_id 
> AS pets_person_id, persons_1.id AS persons_1_id, persons_1.firstname AS 
> persons_1_firstname 
> FROM pets LEFT OUTER JOIN persons AS persons_1 ON persons_1.id = 
> pets.person_id
>
>
> And now the problem: in short, when the library is asked to apply an
> "order by" on some column(s), it uses an utility function[2] to find a
> column given it's name in the particular query, and then it applies an
> `order_by()` the original query[3].
>
> So, appending the following lines to the script above:
>
> from metapensiero.sqlalchemy.proxy.utils import col_by_name
>
> col = col_by_name(query.statement, 'firstname')
>
> sorted_query = query.order_by(col)
> print(str(sorted_query))
>
> I get the following, expected, output under SA 1.4:
>
> SELECT pets.id AS pets_id, pets.name AS pets_name, pets.person_id 
> AS pets_person_id, persons_1.id AS persons_1_id, persons_1.firstname AS 
> persons_1_firstname 
> FROM pets LEFT OUTER JOIN persons AS persons_1 ON persons_1.id = 
> pets.person_id ORDER BY persons_1.firstname
>
> while under SA 2.0.15 I get this:
>
> SELECT pets.id AS pets_id, pets.name AS pets_name, pets.person_id 
> AS pets_person_id, persons_1.id AS persons_1_id, persons_1.firstname AS 
> persons_1_firstname 
> FROM pets LEFT OUTER JOIN persons AS persons_1 ON persons_1.id = 
> pets.person_id ORDER BY persons_2.firstname
>
> that obviously fails, with the following error:
>
> sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such 
> column: persons_2.firstname
>
> Putting a breakpoint just before applying that `.order_by()` and
> inspecting the `col` returned by the function, I cannot spot any
> difference in the object, under both SA I see
>
> (Pdb) p col
> Column('firstname', String(), table=)
>
> Can you shed some light on what I can try to smooth this different
> behaviour?
>
> Thanks a lot in 

Re: [sqlalchemy] connection pooling

[Mike Bayer]

> 
>   File 
> "/Users/apple/.virtualenvs/bofa/lib/python3.11/site-packages/sqlalchemy/engine/base.py",
>  line 1968, in _exec_single_context
> 
> self.dialect.do_execute(
> 
>   File 
> "/Users/apple/.virtualenvs/bofa/lib/python3.11/site-packages/sqlalchemy/engine/default.py",
>  line 920, in do_execute
> 
> cursor.execute(statement, parameters)
> 
>   File 
> "/Users/apple/.virtualenvs/bofa/lib/python3.11/site-packages/oracledb/cursor.py",
>  line 378, in execute
> 
> impl.execute(self)
> 
>   File "src/oracledb/impl/thin/cursor.pyx", line 138, in 
> oracledb.thin_impl.ThinCursorImpl.execute
> 
>   File "src/oracledb/impl/thin/protocol.pyx", line 382, in 
> oracledb.thin_impl.Protocol._process_single_message
> 
>   File "src/oracledb/impl/thin/protocol.pyx", line 383, in 
> oracledb.thin_impl.Protocol._process_single_message
> 
>   File "src/oracledb/impl/thin/protocol.pyx", line 343, in 
> oracledb.thin_impl.Protocol._process_message
> 
>   File "src/oracledb/impl/thin/protocol.pyx", line 321, in 
> oracledb.thin_impl.Protocol._process_message
> 
>   File "src/oracledb/impl/thin/protocol.pyx", line 389, in 
> oracledb.thin_impl.Protocol._receive_packet
> 
>   File "src/oracledb/impl/thin/packet.pyx", line 559, in 
> oracledb.thin_impl.ReadBuffer.receive_packet
> 
>   File "src/oracledb/impl/thin/packet.pyx", line 358, in 
> oracledb.thin_impl.ReadBuffer._receive_packet_helper
> 
>   File "src/oracledb/impl/thin/packet.pyx", line 205, in 
> oracledb.thin_impl.ReadBuffer._get_data_from_socket
> 
>   File 
> "/Users/apple/.virtualenvs/bofa/lib/python3.11/site-packages/oracledb/errors.py",
>  line 118, in _raise_err
> 
> raise exc_type(_Error(message)) from cause
> 
> sqlalchemy.exc.DatabaseError: (oracledb.exceptions.DatabaseError) DPY-4011: 
> the database or network closed the connection
> 
> [SQL: select current_timestamp from dual]
> 
> 
> 
> (Background on this error at: https://sqlalche.me/e/20/4xp6)
> 
> 
> On Thu, 8 Jun 2023 at 22:08, Suraj Shaw  wrote:
>> Hi Mike,
>> 
>> Is it like first conn.close() is closing the pool itself instead of 
>> releasing connection back to pool.
>> 
>> Thanks
>> Suraj
>> 
>> On Thu, 8 Jun 2023 at 18:34, Mike Bayer 
>>  wrote:
>>> __
>>> unknown.  I've run your program exactly as written with SQLAlchemy 2.0.15 
>>> and it succeeds on both queries.  I would advise reaching out to 
>>> https://github.com/oracle/python-oracledb/discussions for debugging help.   
>>> feel free to show them our recipe from 
>>> https://docs.sqlalchemy.org/en/20/dialects/oracle.html#using-oracle-database-resident-connection-pooling-drcp
>>>  for reference.
>>> 
>>> On Thu, Jun 8, 2023, at 7:23 AM, Suraj Shaw wrote:
>>>> Hi Team,
>>>> 
>>>> I am using connection pooling in sqlalchemy using oracle own pool.My 
>>>> script looks like this.
>>>> ```
>>>> from sqlalchemy import create_engine,text
>>>> from sqlalchemy.pool import NullPool
>>>> 
>>>> 
>>>> 
>>>> import oracledb
>>>> 
>>>> 
>>>> pool = oracledb.create_pool(user='XXX', password='XXX',dsn='XXX:1521/XXX', 
>>>> min=1, max=5, increment=1)
>>>> 
>>>> def creator():
>>>> 
>>>> return pool.acquire(cclass="MYAPP",purity=oracledb.PURITY_SELF)
>>>> 
>>>> 
>>>> 
>>>> engine = create_engine("oracle+oracledb://", creator=creator, 
>>>> poolclass=NullPool)
>>>> 
>>>> 
>>>> 
>>>> conn = engine.connect()
>>>> 
>>>> result = (conn.execute(text("select current_timestamp from dual"))) 
>>>> 
>>>> for row in result:#first query
>>>> 
>>>> print(row)
>>>> 
>>>> conn.close()
>>>> 
>>>> 
>>>> 
>>>> conn = engine.connect()
>>>> 
>>>> result = (conn.execute(text("select current_timestamp from dual")))
>>>> 
>>>> for row in result: #second query
>>>> 
>>>> print(row)
>>>> 
>>>> 
>>>> 
>>>> conn.close()
>>>> 
>>>> ```
>>>> 
>>>> 
>>>> 
>>>> Here the first query in runn

Re: [sqlalchemy] connection pooling

[Mike Bayer]

On Thu, Jun 8, 2023, at 12:38 PM, Suraj Shaw wrote:
> Hi Mike,
> 
> Is it like first conn.close() is closing the pool itself instead of releasing 
> connection back to pool.


SQLAlchemy calls the .close() method on the connection itself.  Per oracledb 
docs at 
https://python-oracledb.readthedocs.io/en/latest/user_guide/connection_handling.html#creating-a-connection-pool

"When your application has finished performing all required database 
operations, the pooled connection should be released to make it available for 
other users of the pool. You can do this with `ConnectionPool.release()` 
<https://python-oracledb.readthedocs.io/en/latest/api_manual/connection_pool.html#ConnectionPool.release>
 or `Connection.close()` 
<https://python-oracledb.readthedocs.io/en/latest/api_manual/connection.html#Connection.close>."
  

The error you are showing is also not the error that's raised if the pool is 
closed.  here is a demonstration script:


import oracledb
from sqlalchemy import create_engine, text
from sqlalchemy.pool import NullPool

pool = oracledb.create_pool(
user="scott", password="tiger", dsn="oracle18c/xe",
min=2, max=5, increment=1, threaded=True,
encoding="UTF-8", nencoding="UTF-8"
)

def creator():
return pool.acquire(cclass="MYCLASS", purity=oracledb.PURITY_SELF)

engine = create_engine("oracle+oracledb://", creator=creator, 
poolclass=NullPool)

conn = engine.connect()
print(conn.scalar(text("select 1 from dual")))
conn.close()


conn = engine.connect()
print(conn.scalar(text("select 1 from dual")))
conn.close()

input("press enter to see a closed pool try to connect")
pool.close()
conn = engine.connect()


the first two statements succeed.  then key input is requested.  subsequent to 
that, the error message is explicit: "DPY-1002: connection pool is not open".  
this is not the "DPY-4011: the database or network closed the connection" 
message you refer towards:

$ python test3.py 
1
1
press enter to see a closed pool try to connect
Traceback (most recent call last):
  ...
raise exc_type(_Error(message)) from cause
oracledb.exceptions.InterfaceError: DPY-1002: connection pool is not open




I will reiterate my previous post that there's no action for SQLAlchemy to take 
here as of yet and you should post a discussion on the oracledb discussion 
tracker for assistance on what the error message you have actually means, since 
I dont know.





> 
> Thanks
> Suraj
> 
> On Thu, 8 Jun 2023 at 18:34, Mike Bayer  
> wrote:
>> __
>> unknown.  I've run your program exactly as written with SQLAlchemy 2.0.15 
>> and it succeeds on both queries.  I would advise reaching out to 
>> https://github.com/oracle/python-oracledb/discussions for debugging help.   
>> feel free to show them our recipe from 
>> https://docs.sqlalchemy.org/en/20/dialects/oracle.html#using-oracle-database-resident-connection-pooling-drcp
>>  for reference.
>> 
>> On Thu, Jun 8, 2023, at 7:23 AM, Suraj Shaw wrote:
>>> Hi Team,
>>> 
>>> I am using connection pooling in sqlalchemy using oracle own pool.My script 
>>> looks like this.
>>> ```
>>> from sqlalchemy import create_engine,text
>>> from sqlalchemy.pool import NullPool
>>> 
>>> 
>>> 
>>> import oracledb
>>> 
>>> 
>>> pool = oracledb.create_pool(user='XXX', password='XXX',dsn='XXX:1521/XXX', 
>>> min=1, max=5, increment=1)
>>> 
>>> def creator():
>>> 
>>> return pool.acquire(cclass="MYAPP",purity=oracledb.PURITY_SELF)
>>> 
>>> 
>>> 
>>> engine = create_engine("oracle+oracledb://", creator=creator, 
>>> poolclass=NullPool)
>>> 
>>> 
>>> 
>>> conn = engine.connect()
>>> 
>>> result = (conn.execute(text("select current_timestamp from dual"))) 
>>> 
>>> for row in result:#first query
>>> 
>>> print(row)
>>> 
>>> conn.close()
>>> 
>>> 
>>> 
>>> conn = engine.connect()
>>> 
>>> result = (conn.execute(text("select current_timestamp from dual")))
>>> 
>>> for row in result: #second query
>>> 
>>> print(row)
>>> 
>>> 
>>> 
>>> conn.close()
>>> 
>>> ```
>>> 
>>> 
>>> 
>>> Here the first query in running perfectly when the second time when i am 
>>> doing it then it is rasing the folowing error
>>> 
>>> 
>>> 
>>> sqlalchemy.exc.DatabaseErro

Re: [sqlalchemy] connection pooling

[Mike Bayer]

unknown.  I've run your program exactly as written with SQLAlchemy 2.0.15 and 
it succeeds on both queries.  I would advise reaching out to 
https://github.com/oracle/python-oracledb/discussions for debugging help.   
feel free to show them our recipe from 
https://docs.sqlalchemy.org/en/20/dialects/oracle.html#using-oracle-database-resident-connection-pooling-drcp
 for reference. 

On Thu, Jun 8, 2023, at 7:23 AM, Suraj Shaw wrote:
> Hi Team,
> 
> I am using connection pooling in sqlalchemy using oracle own pool.My script 
> looks like this.
> ```
> from sqlalchemy import create_engine,text
> from sqlalchemy.pool import NullPool
> 
> 
> import oracledb
> 
> pool = oracledb.create_pool(user='XXX', password='XXX',dsn='XXX:1521/XXX', 
> min=1, max=5, increment=1)
> 
> def creator():
> 
> return pool.acquire(cclass="MYAPP",purity=oracledb.PURITY_SELF)
> 
> 
> 
> engine = create_engine("oracle+oracledb://", creator=creator, 
> poolclass=NullPool)
> 
> 
> 
> conn = engine.connect()
> 
> result = (conn.execute(text("select current_timestamp from dual"))) 
> 
> for row in result:#first query
> 
> print(row)
> 
> conn.close()
> 
> 
> 
> conn = engine.connect()
> 
> result = (conn.execute(text("select current_timestamp from dual")))
> 
> for row in result: #second query
> 
> print(row)
> 
> 
> 
> conn.close()
> 
> ```
> 
> 
> 
> Here the first query in running perfectly when the second time when i am 
> doing it then it is rasing the folowing error
> 
> 
> sqlalchemy.exc.DatabaseError: (oracledb.exceptions.DatabaseError) DPY-4011: 
> the database or network closed the connection
> 
> Why is this issue coming.
> 
> Is it because of conn.close(). Here the conn.close() is not releasing 
> connection back to the pool.
> 
> Thanks
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/10098e90-038e-478e-8a7e-db5b8e2f7e07n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/4049c5f7-6369-4251-8de4-b116441fb782%40app.fastmail.com.

Re: [sqlalchemy] Sql constructor

[Mike Bayer]

we can't take on maintenance for new extensions within the SQLAlchemy project 
directly but I would encourage you to release your extension as its own 
library; see https://pypi.org/search/?q=sqlalchemy for other examples

On Tue, Jun 6, 2023, at 5:34 PM, Nir Assaraf wrote:
> Hey guys,
> 
> As a part of a project that I'm working on, Iv'e created a SQLAlchemy 
> statement builder using the ast (abstract syntax trees) library.
> This constructor will parse strings that are valid python logical statements 
> and will convert them to the appropriate SQLAlchemy objects.
> 
> For example:
> The constructor will take a string of the following format and a list of 
> models/modelbase in which the columns exist.
> 
> string_a = '(column_a == 2 or column_b == 3) or column_c in [1,2,3]'
> 
> and returns the SQLAlchemy objects that defines this logic and can be used in 
> a query's where clause.
> 
> This was used to automate query building for a projects with thousands of 
> different logics that where defined by the client.
> 
> I was just wondering if you think there is a place for something like that in 
> SQLAlchemy or related projects.
> 
> Best regards,
> 
> Nir
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/a3617ee6-513f-4d4c-a0bb-04dbf83552c0n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/ac072c46-04f6-48dc-8e73-f6b04b8556ba%40app.fastmail.com.

Re: [sqlalchemy] subquery relationships?

[Mike Bayer]

docs for this are at:

https://docs.sqlalchemy.org/en/20/orm/join_conditions.html

there are several approaches to relationships with subqueries, the most open 
ended is the one described at 
https://docs.sqlalchemy.org/en/20/orm/join_conditions.html#relationship-to-aliased-class



On Wed, May 31, 2023, at 10:13 AM, Dan Stromberg wrote:
> Hi folks.
>
> I have a subquery that is selected from a table with 5 foreign keys,
> and joined with another table with 3 foreign keys.
>
> And then that subquery is used in a join with the table having the 3
> foreign keys again.
>
> I don't know how to tell what column(s) that join is happening on.
>
> What goes into SQLAlchemy's "thought process" when it joins a table
> with a subquery?
>
> Thanks!
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/CAOvKW56oZPu4NByXY1qZO99T8BQ_L54w42y6cex9zJyEoPmejw%40mail.gmail.com.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/4bf74f30-bb42-43b7-a0a1-98ebb4c58a02%40app.fastmail.com.

Re: [sqlalchemy] sqlalchemy returns None for sqlite tables which already have data

[Mike Bayer]

like someone mentioned there, showing the classes doesn't say much.I'd 
check that the table name is correct and definitely matches what's in the 
SQLite database that you are actually accessing, as well as that the primary 
key column in those tables is actually a primary key, or at least not NULL 
values.

On Mon, May 22, 2023, at 8:53 AM, m7mđ ĕğý wrote:
> Hello mates,
> 
> As my last msg had a so bad format i will keep the link of my original 
> question on  stackoverflow   :
> 
> https://stackoverflow.com/questions/76304295/sqlalchemy-returns-none-for-sqlite-tables-which-already-have-data
>  
> 
> i hope i get answers from you mates.
> 
> Thanks
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/530d0f09-8624-4f38-9a29-711a79a1d6d1n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/92475a97-d14e-4751-8f03-afa3f1b6f485%40app.fastmail.com.

Re: [sqlalchemy] Inconsistent SQL Generation in 2.0

[Mike Bayer]

On Mon, May 15, 2023, at 1:28 PM, Benjamin Taub wrote:
> Hey, Mike!
> I just wanted to let you know I figured this out. Your sample code led me to 
> something I hadn't considered. After the routine I was stuck in, I take the 
> SQLAlchemy-generated code, turn it into a string, and add some 
> manually-generated SQL. I used a command like *str(SQL)* to get this string. 

yes, it was fairly clear you were likely doing something like this.


> 
> 
> 
> I am pretty sure that this worked in v1.4.

in only "worked" out of luck, that the generic SQL happened to match what MySQL 
wanted.   There are many points at which this would fail with SQL constructs 
that have some idiosyncratic rendering in MySQL.

That said, it's not exactly familiar to me how identifiers that start with 
numbers started getting quoted for all dialects; there was definitely something 
there regarding the Oracle backend but I dont know offhand if/how that got 
generalized.


> 
> 
> 
> However, it turns out that I now have to use 
> *str(sql.compile(common.qry_engine))*.

this is definitely the way to do this for stringifying SQL, as every backend 
has lots of idiosyncracies in its SQL format.

now there is a bigger issue which is that it's not the best way to add text to 
SQL by running str() on it, when you run str(stmt) and then just run the text, 
a whole lot of valuable internal data is lost when you execute the statement 
which helps with things like applying the correct types to result rows, 
fetching ORM objects, etc. The str() process is also not that cheap as 
there is no caching used, running statements directly allows the engine to take 
advantage of SQL caching.

There's other ways to add literal strings inside of SQL, with two general 
categories of injection of strings as the object is constructed, and mutation 
of the final compiled SQL string as it is executed.  depending on what you are 
doing, one or the other approach is appropriate and will retain all the 
advantages of compiled execution and caching.


> 

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/aa9d5fc2-fedf-46b0-95c5-ee58dc2372c0%40app.fastmail.com.

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

thanks!

On Sun, May 14, 2023, at 6:59 PM, Shane Holcombe wrote:
> I've tested all the ways I could previously trigger the issue and still, no 
> Nones seem to even be produced to be filtered out, that fix definitely seems 
> to have sorted the issue out
> 
> On Sunday, May 14, 2023 at 7:42:54 AM UTC+10 Shane Holcombe wrote:
>> Yes so I'm testing the second solution with a breakpoint on the area that 
>> should filter out those Nones, I wanted to see wether the Nones are being 
>> removed or are no longer showing up. So far I haven't seen that code be 
>> used, which is great, means the race condition thing must have helped. I 
>> can't be 100% sure as it was hard to trigger but I haven't managed to just 
>> yet and I've tried what usually caused the issue. I'll keep testing some 
>> more and let you know if it shows up again but this seems much better, thank 
>> so much.
>> 
>> On Sunday, May 14, 2023 at 2:37:37 AM UTC+10 Mike Bayer wrote:
>>> __
>>> the second solution has not only the removal of where the theoretical "race 
>>> condition" might happen, but it also removes the "None" entries in any 
>>> case, so even if the "None" thing is coming from somewhere else, they won't 
>>> make it to the select() statement at the end.  
>>> 
>>> On Sat, May 13, 2023, at 3:36 AM, Shane Holcombe wrote:
>>>> I've tested both solutions, the first one, moving the if c is None: return 
>>>> down a few lines seems to work and the generated SQL still seems to be 
>>>> correct in the instances where that code is triggered.
>>>> From some early testing as well the other solution seems to work as well, 
>>>> it's hard to test 100% as this was always a bit troublesome to replicate, 
>>>> I had a series of events that seemed to trigger it somewhat regularly and 
>>>> it hasn't yet.
>>>> The idea this might be a race condition does make sense as the model that 
>>>> seems to more often than not return the missing columns is used basically 
>>>> everywhere and will be joined to from other models very frequently, often 
>>>> multiple times from the same query, with multiple different queries on a 
>>>> page load, the 'xx_application_user' model in the query I sent.
>>>> I'll test it again a few more times/ways later tonight/tomorrow morning 
>>>> when I get a chance again and let you know
>>>> Thanks so much
>>>> On Saturday, May 13, 2023 at 1:50:56 PM UTC+10 Mike Bayer wrote:
>>>>> __
>>>>> in the below mentioned issue I've created a patch at 
>>>>> https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4615 that removes 
>>>>> a thread-sensitive approach to generating an internal structure, which is 
>>>>> where I suspect this might be happening (if anywhere), and then it also 
>>>>> adds a guard against the None objects in any case.you'd want to take 
>>>>> a look at that and see if it resolves.
>>>>> 
>>>>> On Fri, May 12, 2023, at 11:30 PM, Mike Bayer wrote:
>>>>>> I've opened https://github.com/sqlalchemy/sqlalchemy/issues/9777 for 
>>>>>> this.
>>>>>> 
>>>>>> I have another theory where something might be going wrong earler in the 
>>>>>> chain. This change would be "simpler" in that it removes something 
>>>>>> that's complicated, prone to race conditions hence seems a good 
>>>>>> possibility of being involved here.   This would be the patch:
>>>>>> 
>>>>>> diff --git a/lib/sqlalchemy/orm/strategies.py 
>>>>>> b/lib/sqlalchemy/orm/strategies.py
>>>>>> index 8e06c4f598..bd0193b905 100644
>>>>>> --- a/lib/sqlalchemy/orm/strategies.py
>>>>>> +++ b/lib/sqlalchemy/orm/strategies.py
>>>>>> @@ -2324,8 +2324,7 @@ class JoinedLoader(AbstractRelationshipLoader):
>>>>>>  else:
>>>>>>  context.attributes[key] = idx = context.attributes[key] + 1
>>>>>>  
>>>>>> -if idx >= len(self._aliased_class_pool):
>>>>>> -to_adapt = orm_util.AliasedClass(
>>>>>> +return orm_util.AliasedClass(
>>>>>>  self.mapper,
>>>>>>  alias=alt_selectable._anonymous_fromclause(flat=True)
>>>>>>  if alt_selectable is not None
>>>>&g

Re: [sqlalchemy] Inconsistent SQL Generation in 2.0

[Mike Bayer]

On Sun, May 14, 2023, at 12:39 PM, Benjamin Taub wrote:
> Hi, Mike!
> I tried your code, and it worked after I set the dialect to mysql+pymysql.
> 
> Given this, in my case, I believe the problem stems from the fact that I am 
> starting with a generic SELECT call that isn't moored to a dialect. I start 
> with 
> *sql = select()*

Modern SQLAlchemy SQL constructs like select() have no connection to any 
dialect.Only if you are using 1.x and using the very dated "bound metadata" 
concept which is now removed, is there any truth to this.   But any SQL 
construct can be compiled with any dialect at all times (Assuming it contains 
no dialect-specific constructs).

> 
> Which I have now changed to (self.tbl is a sqlalchemy table object attached 
> to MySQL)...
> *sql = self.tbl.select()
*
> The problem now is that I have a null column object with a label that isn't 
> attached to self.tbl, and I can't figure out how to tell it to generate as 
> MySQL...
> tbl_col = null().label(tmplt_col['name'])
>  
> It is tbl_col that eventually gives me the quote problem in SQL generation. 
> Do you have any ideas for me on how to have this generate in the right 
> dialect? Am I going about this the wrong way?

I would need to see an example of what you're doing as this does not really 
indicate why there would be any problem.   as mentioned previously, any SQL 
construct when compiled is always given a dialect with which to compile with, 
and you can always pass this in.

it's still not clear here if you are using .compile() directly or not as I dont 
really have an understanding of what you're doing.


> 
> Thanks again for your help!
> Ben
> On Saturday, May 13, 2023 at 11:33:03 PM UTC-4 Mike Bayer wrote:
>> 
>> 
>> On Sat, May 13, 2023, at 5:12 PM, Benjamin Taub wrote:
>>> Thank you, Mike, but aren't I using the correct dialect with this 
>>> create_engine() command?
>>> 
>>> *qry_engine = create_engine('mysql+pymysql://' + db_uid + ':' + 
>>> db_pw + '@' + db_addr, connect_args=connect_args,
>>>pool_recycle=3600, echo=False, 
>>> future=True)*
>>> 
>>> Or, am I missing something? (BTW, I'm using core, not ORM).
>> 
>> looks fine.  I would need to see a complete example in how you are seeing it 
>> generate quotes.  if you try the program I attached previously, you should 
>> note that it generates backticks.
>> 
>>> 
>>> Thanks again!
>>> Ben 
>>> On Friday, May 12, 2023 at 4:48:45 PM UTC-4 Mike Bayer wrote:
>>>> 
>>>> 
>>>> On Fri, May 12, 2023, at 4:30 PM, Benjamin Taub wrote:
>>>>> I have code that worked under SQLAlchemy 1.4 but recently upgraded to 2. 
>>>>> I am using the add_columns() method to add columns to an existing SQL 
>>>>> statement. The resultant queries sometimes, but not always, crash. I 
>>>>> believe the issue happens when the schema/database name (I'm using MySQL) 
>>>>> starts with a number. When the schema name starts with a letter, the 
>>>>> result runs fine. However, when it starts with a number, the query 
>>>>> double-quotes the schema name, causing the query to crash.
>>>>> 
>>>>> Here is an example...
>>>>> My code: *sql = sql.add_columns(self.tbl.c[field])*
>>>>> 
>>>>> When the schema holding self.tbl.c[field] starts with a letter 
>>>>> (c6961a19b7ed031ce902f056c725b3e3), the following SQL is generated:
>>>>> *SELECT NULL AS "Application Id", NULL AS "First Name", NULL AS "Last 
>>>>> Name", NULL AS "Email", 
>>>>> c6961a19b7ed031ce902f056c725b3e3.t_31392eb2e6980f4d5082b7861182f2b4.master_key
>>>>>  
>>>>> FROM c6961a19b7ed031ce902f056c725b3e3.t_31392eb2e6980f4d5082b7861182f2b4*
>>>>> 
>>>>> However, when the schema name starts with a number 
>>>>> (283ac7717fe770c5ed6d425c0c739cba), the following SQL results:
>>>>> *SELECT NULL AS "Application Id", NULL AS "First Name", NULL AS "Last 
>>>>> Name", NULL AS "Email", 
>>>>> "283ac7717fe770c5ed6d425c0c739cba".t_59a33cbea3617986d810e9fbae60ba19.master_key
>>>>>  
>>>>> FROM 
>>>>> "283ac7717fe770c5ed6d425c0c739cba".t_59a33cbea3617986d810e9fbae60ba19*
>>>>> 
>>>>> Note the double quotes around the schema name. This second SQL crashes as 
>>&g

Re: [sqlalchemy] Inconsistent SQL Generation in 2.0

[Mike Bayer]

On Sat, May 13, 2023, at 5:12 PM, Benjamin Taub wrote:
> Thank you, Mike, but aren't I using the correct dialect with this 
> create_engine() command?
> 
> *qry_engine = create_engine('mysql+pymysql://' + db_uid + ':' + db_pw 
> + '@' + db_addr, connect_args=connect_args,
>pool_recycle=3600, echo=False, 
> future=True)*
> 
> Or, am I missing something? (BTW, I'm using core, not ORM).

looks fine.  I would need to see a complete example in how you are seeing it 
generate quotes.  if you try the program I attached previously, you should note 
that it generates backticks.

> Thanks again!
> Ben 
> On Friday, May 12, 2023 at 4:48:45 PM UTC-4 Mike Bayer wrote:
>> 
>> 
>> On Fri, May 12, 2023, at 4:30 PM, Benjamin Taub wrote:
>>> I have code that worked under SQLAlchemy 1.4 but recently upgraded to 2. I 
>>> am using the add_columns() method to add columns to an existing SQL 
>>> statement. The resultant queries sometimes, but not always, crash. I 
>>> believe the issue happens when the schema/database name (I'm using MySQL) 
>>> starts with a number. When the schema name starts with a letter, the result 
>>> runs fine. However, when it starts with a number, the query double-quotes 
>>> the schema name, causing the query to crash.
>>> 
>>> Here is an example...
>>> My code: *sql = sql.add_columns(self.tbl.c[field])*
>>> 
>>> When the schema holding self.tbl.c[field] starts with a letter 
>>> (c6961a19b7ed031ce902f056c725b3e3), the following SQL is generated:
>>> *SELECT NULL AS "Application Id", NULL AS "First Name", NULL AS "Last 
>>> Name", NULL AS "Email", 
>>> c6961a19b7ed031ce902f056c725b3e3.t_31392eb2e6980f4d5082b7861182f2b4.master_key
>>>  
>>> FROM c6961a19b7ed031ce902f056c725b3e3.t_31392eb2e6980f4d5082b7861182f2b4*
>>> 
>>> However, when the schema name starts with a number 
>>> (283ac7717fe770c5ed6d425c0c739cba), the following SQL results:
>>> *SELECT NULL AS "Application Id", NULL AS "First Name", NULL AS "Last 
>>> Name", NULL AS "Email", 
>>> "283ac7717fe770c5ed6d425c0c739cba".t_59a33cbea3617986d810e9fbae60ba19.master_key
>>>  
>>> FROM "283ac7717fe770c5ed6d425c0c739cba".t_59a33cbea3617986d810e9fbae60ba19*
>>> 
>>> Note the double quotes around the schema name. This second SQL crashes as 
>>> invalid. Back quotes (`) would probably work fine in this situation, and 
>>> could be helpful, but double quotes (") are, I think, the cause of my 
>>> problem.
>>> 
>>> Is there some parameter or assumption that I'm not understanding, or did I 
>>> find a bug?
>> 
>> The quoting, if it were the correct quoting format, should be fine.   As to 
>> why it's the quote char and not the backtick, are you compiling these 
>> queries manually?You would want to make sure a MySQL dialect is in use, 
>> which would be using backticks for quoting, unless that dialect were 
>> initialized against a MySQL database that has ANSI_QUOTES set.
>> 
>> TL;DR quoting is a new thing here but SQLAlchemy should render the correct 
>> quotes when used with the MySQL dialect.
>> 
>> Here's a demo:
>> 
>> from sqlalchemy import Column
>> from sqlalchemy import create_engine
>> from sqlalchemy import Integer
>> from sqlalchemy import select
>> from sqlalchemy import String
>> from sqlalchemy.orm import declarative_base
>> Base = declarative_base()
>> 
>> 
>> class A(Base):
>> __tablename__ = 't_59a33cbea3617986d810e9fbae60ba19'
>> 
>> __table_args__ = {
>> "schema": "283ac7717fe770c5ed6d425c0c739cba"
>> }
>> id = Column(Integer, primary_key=True)
>> data = Column(String)
>> 
>> e = create_engine("mysql://")
>> 
>> stmt = select(A)
>> 
>> print(stmt.compile(e))
>> 
>> output:
>> 
>> SELECT 
>> `283ac7717fe770c5ed6d425c0c739cba`.t_59a33cbea3617986d810e9fbae60ba19.id, 
>> `283ac7717fe770c5ed6d425c0c739cba`.t_59a33cbea3617986d810e9fbae60ba19.data 
>> FROM `283ac7717fe770c5ed6d425c0c739cba`.t_59a33cbea3617986d810e9fbae60ba19
>> 
>> 
>> 
>>> 
>>> Thank you!
>>> Ben
>>> 
>>> 
>>> 
>>> 
>>> --
>>> SQLAlchemy -
>>> The Python SQL Toolkit and Object Relational Mapper
>>>  
>>> http://www.sqlalchemy.org/
&g

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

the second solution has not only the removal of where the theoretical "race 
condition" might happen, but it also removes the "None" entries in any case, so 
even if the "None" thing is coming from somewhere else, they won't make it to 
the select() statement at the end.   

On Sat, May 13, 2023, at 3:36 AM, Shane Holcombe wrote:
> I've tested both solutions, the first one, moving the if c is None: return 
> down a few lines seems to work and the generated SQL still seems to be 
> correct in the instances where that code is triggered.
> From some early testing as well the other solution seems to work as well, 
> it's hard to test 100% as this was always a bit troublesome to replicate, I 
> had a series of events that seemed to trigger it somewhat regularly and it 
> hasn't yet.
> The idea this might be a race condition does make sense as the model that 
> seems to more often than not return the missing columns is used basically 
> everywhere and will be joined to from other models very frequently, often 
> multiple times from the same query, with multiple different queries on a page 
> load, the 'xx_application_user' model in the query I sent.
> I'll test it again a few more times/ways later tonight/tomorrow morning when 
> I get a chance again and let you know
> Thanks so much
> On Saturday, May 13, 2023 at 1:50:56 PM UTC+10 Mike Bayer wrote:
>> __
>> in the below mentioned issue I've created a patch at 
>> https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4615 that removes a 
>> thread-sensitive approach to generating an internal structure, which is 
>> where I suspect this might be happening (if anywhere), and then it also adds 
>> a guard against the None objects in any case.you'd want to take a look 
>> at that and see if it resolves.
>> 
>> On Fri, May 12, 2023, at 11:30 PM, Mike Bayer wrote:
>>> I've opened https://github.com/sqlalchemy/sqlalchemy/issues/9777 for this.
>>> 
>>> I have another theory where something might be going wrong earler in the 
>>> chain. This change would be "simpler" in that it removes something 
>>> that's complicated, prone to race conditions hence seems a good possibility 
>>> of being involved here.   This would be the patch:
>>> 
>>> diff --git a/lib/sqlalchemy/orm/strategies.py 
>>> b/lib/sqlalchemy/orm/strategies.py
>>> index 8e06c4f598..bd0193b905 100644
>>> --- a/lib/sqlalchemy/orm/strategies.py
>>> +++ b/lib/sqlalchemy/orm/strategies.py
>>> @@ -2324,8 +2324,7 @@ class JoinedLoader(AbstractRelationshipLoader):
>>>  else:
>>>  context.attributes[key] = idx = context.attributes[key] + 1
>>>  
>>> -if idx >= len(self._aliased_class_pool):
>>> -to_adapt = orm_util.AliasedClass(
>>> +return orm_util.AliasedClass(
>>>  self.mapper,
>>>  alias=alt_selectable._anonymous_fromclause(flat=True)
>>>  if alt_selectable is not None
>>> @@ -2334,14 +2333,6 @@ class JoinedLoader(AbstractRelationshipLoader):
>>>  use_mapper_path=True,
>>>  )
>>>  
>>> -# load up the .columns collection on the Alias() before
>>> -# the object becomes shared among threads.  this prevents
>>> -# races for column identities.
>>> -inspect(to_adapt).selectable.c
>>> -self._aliased_class_pool.append(to_adapt)
>>> -
>>> -return self._aliased_class_pool[idx]
>>> -
>>>  def _generate_row_adapter(
>>>  self,
>>>  compile_state,
>>> 
>>> 
>>> 
>>> 
>>> On Fri, May 12, 2023, at 11:05 PM, Shane Holcombe wrote:
>>>> There seems to be a few moving parts to this so it's been hard to track 
>>>> down, I can give that fix a try in a few hours when I'm home.
>>>> I was worried something like that might lose some columns but I'll try and 
>>>> see what happens.
>>>> Thanks
>>>> On Saturday, May 13, 2023 at 12:59:33 PM UTC+10 Mike Bayer wrote:
>>>>> __
>>>>> well, not sure yet how a None is coming in there that is sporadic, or 
>>>>> even at all, but if that's what's happening then we would think this 
>>>>> patch would fix the problem, the Q is does the query still produce the 
>>>>> correct results:
>>>>> 
>>>>> diff --git a/lib/sqlalchemy/orm/strategies.py 
>>>>> b/lib/sqlalchemy/orm/strategie

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

in the below mentioned issue I've created a patch at 
https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4615 that removes a 
thread-sensitive approach to generating an internal structure, which is where I 
suspect this might be happening (if anywhere), and then it also adds a guard 
against the None objects in any case.you'd want to take a look at that and 
see if it resolves.

On Fri, May 12, 2023, at 11:30 PM, Mike Bayer wrote:
> I've opened https://github.com/sqlalchemy/sqlalchemy/issues/9777 for this.
> 
> I have another theory where something might be going wrong earler in the 
> chain. This change would be "simpler" in that it removes something that's 
> complicated, prone to race conditions hence seems a good possibility of being 
> involved here.   This would be the patch:
> 
> diff --git a/lib/sqlalchemy/orm/strategies.py 
> b/lib/sqlalchemy/orm/strategies.py
> index 8e06c4f598..bd0193b905 100644
> --- a/lib/sqlalchemy/orm/strategies.py
> +++ b/lib/sqlalchemy/orm/strategies.py
> @@ -2324,8 +2324,7 @@ class JoinedLoader(AbstractRelationshipLoader):
>  else:
>  context.attributes[key] = idx = context.attributes[key] + 1
>  
> -if idx >= len(self._aliased_class_pool):
> -to_adapt = orm_util.AliasedClass(
> +return orm_util.AliasedClass(
>  self.mapper,
>  alias=alt_selectable._anonymous_fromclause(flat=True)
>  if alt_selectable is not None
> @@ -2334,14 +2333,6 @@ class JoinedLoader(AbstractRelationshipLoader):
>  use_mapper_path=True,
>  )
>  
> -# load up the .columns collection on the Alias() before
> -# the object becomes shared among threads.  this prevents
> -# races for column identities.
> -inspect(to_adapt).selectable.c
> -self._aliased_class_pool.append(to_adapt)
> -
> -return self._aliased_class_pool[idx]
> -
>  def _generate_row_adapter(
>  self,
>  compile_state,
> 
> 
> 
> 
> On Fri, May 12, 2023, at 11:05 PM, Shane Holcombe wrote:
>> There seems to be a few moving parts to this so it's been hard to track 
>> down, I can give that fix a try in a few hours when I'm home.
>> I was worried something like that might lose some columns but I'll try and 
>> see what happens.
>> Thanks
>> On Saturday, May 13, 2023 at 12:59:33 PM UTC+10 Mike Bayer wrote:
>>> __
>>> well, not sure yet how a None is coming in there that is sporadic, or even 
>>> at all, but if that's what's happening then we would think this patch would 
>>> fix the problem, the Q is does the query still produce the correct results:
>>> 
>>> diff --git a/lib/sqlalchemy/orm/strategies.py 
>>> b/lib/sqlalchemy/orm/strategies.py
>>> index 8e06c4f598..2bac1aad48 100644
>>> --- a/lib/sqlalchemy/orm/strategies.py
>>> +++ b/lib/sqlalchemy/orm/strategies.py
>>> @@ -218,10 +218,10 @@ class ColumnLoader(LoaderStrategy):
>>>  if adapter:
>>>  if check_for_adapt:
>>>  c = adapter.adapt_check_present(c)
>>> -if c is None:
>>> -return
>>>      else:
>>>  c = adapter.columns[c]
>>> +if c is None:
>>> +return
>>>  
>>>  compile_state._append_dedupe_col_collection(c, 
>>> column_collection)
>>> 
>>> 
>>> 
>>> can you try that patch ?
>>> 
>>> 
>>> On Fri, May 12, 2023, at 10:47 PM, Mike Bayer wrote:
>>>> OK, maybe you are onto something with the theory re: 
>>>> JoinedLoader._generate_row_adapter(). will look at that
>>>> 
>>>> On Fri, May 12, 2023, at 6:16 PM, Shane Holcombe wrote:
>>>>> Thanks for linking that github issue, completely missed that one.
>>>>> 
>>>>> Here's the complete stack trace, sorry for not including that at the start
>>>>> 
>>>>> Traceback (most recent call last):
>>>>>   File 
>>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py", 
>>>>> line 333, in run_wsgi
>>>>> execute(self.server.app)
>>>>>   File 
>>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py", 
>>>>> line 320, in execute
>>>>> application_iter = app(environ, start_response)
>>>>>   File "/Users/sf

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

I've opened https://github.com/sqlalchemy/sqlalchemy/issues/9777 for this.

I have another theory where something might be going wrong earler in the chain. 
This change would be "simpler" in that it removes something that's 
complicated, prone to race conditions hence seems a good possibility of being 
involved here.   This would be the patch:

diff --git a/lib/sqlalchemy/orm/strategies.py b/lib/sqlalchemy/orm/strategies.py
index 8e06c4f598..bd0193b905 100644
--- a/lib/sqlalchemy/orm/strategies.py
+++ b/lib/sqlalchemy/orm/strategies.py
@@ -2324,8 +2324,7 @@ class JoinedLoader(AbstractRelationshipLoader):
 else:
 context.attributes[key] = idx = context.attributes[key] + 1
 
-if idx >= len(self._aliased_class_pool):
-to_adapt = orm_util.AliasedClass(
+return orm_util.AliasedClass(
 self.mapper,
 alias=alt_selectable._anonymous_fromclause(flat=True)
 if alt_selectable is not None
@@ -2334,14 +2333,6 @@ class JoinedLoader(AbstractRelationshipLoader):
 use_mapper_path=True,
 )
 
-# load up the .columns collection on the Alias() before
-# the object becomes shared among threads.  this prevents
-# races for column identities.
-inspect(to_adapt).selectable.c
-self._aliased_class_pool.append(to_adapt)
-
-return self._aliased_class_pool[idx]
-
 def _generate_row_adapter(
 self,
 compile_state,




On Fri, May 12, 2023, at 11:05 PM, Shane Holcombe wrote:
> There seems to be a few moving parts to this so it's been hard to track down, 
> I can give that fix a try in a few hours when I'm home.
> I was worried something like that might lose some columns but I'll try and 
> see what happens.
> Thanks
> On Saturday, May 13, 2023 at 12:59:33 PM UTC+10 Mike Bayer wrote:
>> __
>> well, not sure yet how a None is coming in there that is sporadic, or even 
>> at all, but if that's what's happening then we would think this patch would 
>> fix the problem, the Q is does the query still produce the correct results:
>> 
>> diff --git a/lib/sqlalchemy/orm/strategies.py 
>> b/lib/sqlalchemy/orm/strategies.py
>> index 8e06c4f598..2bac1aad48 100644
>> --- a/lib/sqlalchemy/orm/strategies.py
>> +++ b/lib/sqlalchemy/orm/strategies.py
>> @@ -218,10 +218,10 @@ class ColumnLoader(LoaderStrategy):
>>  if adapter:
>>  if check_for_adapt:
>>  c = adapter.adapt_check_present(c)
>> -if c is None:
>> -return
>>  else:
>>  c = adapter.columns[c]
>> +if c is None:
>> +return
>>  
>>      compile_state._append_dedupe_col_collection(c, 
>> column_collection)
>> 
>> 
>> 
>> can you try that patch ?
>> 
>> 
>> On Fri, May 12, 2023, at 10:47 PM, Mike Bayer wrote:
>>> OK, maybe you are onto something with the theory re: 
>>> JoinedLoader._generate_row_adapter(). will look at that
>>> 
>>> On Fri, May 12, 2023, at 6:16 PM, Shane Holcombe wrote:
>>>> Thanks for linking that github issue, completely missed that one.
>>>> 
>>>> Here's the complete stack trace, sorry for not including that at the start
>>>> 
>>>> Traceback (most recent call last):
>>>>   File 
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py", 
>>>> line 333, in run_wsgi
>>>> execute(self.server.app)
>>>>   File 
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py", 
>>>> line 320, in execute
>>>> application_iter = app(environ, start_response)
>>>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", 
>>>> line 2551, in __call__
>>>> return self.wsgi_app(environ, start_response)
>>>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", 
>>>> line 2531, in wsgi_app
>>>> response = self.handle_exception(e)
>>>>   File 
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_cors/extension.py", 
>>>> line 165, in wrapped_function
>>>> return cors_after_request(app.make_response(f(*args, **kwargs)))
>>>>   File 
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", line 
>>>> 673, in error_router
>>>> return original_handler(f)
>>>>   File 
>&g

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

well, not sure yet how a None is coming in there that is sporadic, or even at 
all, but if that's what's happening then we would think this patch would fix 
the problem, the Q is does the query still produce the correct results:

diff --git a/lib/sqlalchemy/orm/strategies.py b/lib/sqlalchemy/orm/strategies.py
index 8e06c4f598..2bac1aad48 100644
--- a/lib/sqlalchemy/orm/strategies.py
+++ b/lib/sqlalchemy/orm/strategies.py
@@ -218,10 +218,10 @@ class ColumnLoader(LoaderStrategy):
 if adapter:
 if check_for_adapt:
 c = adapter.adapt_check_present(c)
-if c is None:
-return
 else:
 c = adapter.columns[c]
+if c is None:
+return
 
 compile_state._append_dedupe_col_collection(c, column_collection)



can you try that patch ?


On Fri, May 12, 2023, at 10:47 PM, Mike Bayer wrote:
> OK, maybe you are onto something with the theory re: 
> JoinedLoader._generate_row_adapter(). will look at that
> 
> On Fri, May 12, 2023, at 6:16 PM, Shane Holcombe wrote:
>> Thanks for linking that github issue, completely missed that one.
>> 
>> Here's the complete stack trace, sorry for not including that at the start
>> 
>> Traceback (most recent call last):
>>   File 
>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py", line 
>> 333, in run_wsgi
>> execute(self.server.app)
>>   File 
>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py", line 
>> 320, in execute
>> application_iter = app(environ, start_response)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", line 
>> 2551, in __call__
>> return self.wsgi_app(environ, start_response)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", line 
>> 2531, in wsgi_app
>> response = self.handle_exception(e)
>>   File 
>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_cors/extension.py", 
>> line 165, in wrapped_function
>> return cors_after_request(app.make_response(f(*args, **kwargs)))
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", 
>> line 673, in error_router
>> return original_handler(f)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", 
>> line 671, in error_router
>> return self.handle_error(e)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", line 
>> 2528, in wsgi_app
>> response = self.full_dispatch_request()
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", line 
>> 1825, in full_dispatch_request
>> rv = self.handle_user_exception(e)
>>   File 
>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_cors/extension.py", 
>> line 165, in wrapped_function
>> return cors_after_request(app.make_response(f(*args, **kwargs)))
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", 
>> line 673, in error_router
>> return original_handler(f)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", 
>> line 671, in error_router
>> return self.handle_error(e)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", line 
>> 1823, in full_dispatch_request
>> rv = self.dispatch_request()
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py", line 
>> 1799, in dispatch_request
>> return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", 
>> line 404, in wrapper
>> resp = resource(*args, **kwargs)
>>   File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/views.py", 
>> line 107, in view
>> return current_app.ensure_sync(self.dispatch_request)(**kwargs)
>>   File 
>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/resource.py", 
>> line 46, in dispatch_request
>> resp = meth(*args, **kwargs)
>>   File 
>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_cors/decorator.py", 
>> line 128, in wrapped_function
>> resp = make_response(f(*args, **kwargs))
>>   File "/Users/sfh/PycharmProjects/vita/stockholm/geneva/api/auth.py", line 
>> 151, in decorated
>> return f(*args, **kwargs)
>>   File 
>> "/Users/sfh/PycharmProjects

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

python3.9/site-packages/sqlalchemy/sql/elements.py", 
> line 483, in _execute_on_connection
> return connection._execute_clauseelement(
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/engine/base.py", 
> line 1629, in _execute_clauseelement
> compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", 
> line 671, in _compile_w_cache
> compiled_sql = self._compiler(
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", 
> line 288, in _compiler
> return dialect.statement_compiler(dialect, self, **kw)
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", 
> line 1426, in __init__
> Compiled.__init__(self, dialect, statement, **kwargs)
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", 
> line 867, in __init__
> self.string = self.process(self.statement, **compile_kwargs)
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", 
> line 912, in process
> return obj._compiler_dispatch(self, **kwargs)
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", 
> line 143, in _compiler_dispatch
> return meth(self, **kw)  # type: ignore  # noqa: E501
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", 
> line 4593, in visit_select
> compile_state = select_stmt._compile_state_factory(
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/base.py", 
> line 678, in create_for_statement
> return klass.create_for_statement(statement, compiler, **kw)
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/orm/context.py", 
> line 1101, in create_for_statement
> SelectState.__init__(self, self.statement, compiler, **kw)
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py",
>  line 4501, in __init__
> self.froms = self._get_froms(statement)
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py",
>  line 4599, in _get_froms
> [
>   File 
> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py",
>  line 4600, in 
> element._from_objects
> AttributeError: 'NoneType' object has no attribute '_from_objects'
> In this case the statement we use, extended_lead_for_id, is
> 
> from geneva.models import person_factory
> stmt = select(Lead).where(Lead.id == lead_id).options(
> joinedload(Lead.lead_person_client),
> joinedload(Lead.lead_person_primary_contact),
> joinedload(Lead.person),
> joinedload(Lead.person).joinedload(person_factory.Person.contract),
> joinedload(Lead.assigned_to_application_user),
> joinedload(Lead.in_progress_by_application_user),
> joinedload(Lead.closed_by_application_user),
> joinedload(Lead.lu_client_type),
> joinedload(Lead.lu_lead_origin_type),
> joinedload(Lead.lu_lead_status),
> joinedload(Lead.lu_lead_closed_state),
> joinedload(Lead.lu_lead_lost_reason),
> joinedload(Lead.lu_marketing_channel),
> joinedload(Lead.lu_marketing_channel_digital),
> joinedload(Lead.lu_marketing_channel_on_site),
> joinedload(Lead.lu_marketing_channel_out_of_home),
> joinedload(Lead.lu_marketing_channel_print),
> joinedload(Lead.lu_marketing_channel_radio),
> joinedload(Lead.lu_marketing_channel_referral),
> joinedload(Lead.lu_marketing_channel_tv),
> joinedload(Lead.lu_marketing_channel_paid_directory),
> joinedload(Lead.lu_marketing_channel_direct_mail),
> joinedload(Lead.referral_supplier),
> joinedload(Lead.council),
> joinedload(Lead.hospital),
> )
> return db.session.execute(stmt).scalar_one_or_none()
> 
> As for the models, we don't really use any default loaders on relationship 
> creation, we've tended to avoid that and put the joins in the select 
> statements we create
> If you really need to see a particular model from this let me know.
> 
> Thanks again for the help
> On Saturday, May 13, 2023 at 1:28:27 AM UTC+10 Mike Bayer wrote:
>> __
>> I really need to see:
>> 
>> 1. actual models to the degree they illustrate the default loaders set up on 
>> relationships
>> 2. query in use
>> 3. **complete** stack trace, put it on a github gist if it's too long.  
>> every line, the error, top to bottom.  thanks
>> 
>> On Fri, May 12, 2023, at 11:25 AM, Mike Bayer wrote:
>>> 
>>> 
>>> On Fri, May 12, 2023, 

Re: [sqlalchemy] Inconsistent SQL Generation in 2.0

[Mike Bayer]

On Fri, May 12, 2023, at 4:30 PM, Benjamin Taub wrote:
> I have code that worked under SQLAlchemy 1.4 but recently upgraded to 2. I am 
> using the add_columns() method to add columns to an existing SQL statement. 
> The resultant queries sometimes, but not always, crash. I believe the issue 
> happens when the schema/database name (I'm using MySQL) starts with a number. 
> When the schema name starts with a letter, the result runs fine. However, 
> when it starts with a number, the query double-quotes the schema name, 
> causing the query to crash.
> 
> Here is an example...
> My code: *sql = sql.add_columns(self.tbl.c[field])*
> 
> When the schema holding self.tbl.c[field] starts with a letter 
> (c6961a19b7ed031ce902f056c725b3e3), the following SQL is generated:
> *SELECT NULL AS "Application Id", NULL AS "First Name", NULL AS "Last Name", 
> NULL AS "Email", 
> c6961a19b7ed031ce902f056c725b3e3.t_31392eb2e6980f4d5082b7861182f2b4.master_key
>  
> FROM c6961a19b7ed031ce902f056c725b3e3.t_31392eb2e6980f4d5082b7861182f2b4*
> 
> However, when the schema name starts with a number 
> (283ac7717fe770c5ed6d425c0c739cba), the following SQL results:
> *SELECT NULL AS "Application Id", NULL AS "First Name", NULL AS "Last Name", 
> NULL AS "Email", 
> "283ac7717fe770c5ed6d425c0c739cba".t_59a33cbea3617986d810e9fbae60ba19.master_key
>  
> FROM "283ac7717fe770c5ed6d425c0c739cba".t_59a33cbea3617986d810e9fbae60ba19*
> 
> Note the double quotes around the schema name. This second SQL crashes as 
> invalid. Back quotes (`) would probably work fine in this situation, and 
> could be helpful, but double quotes (") are, I think, the cause of my problem.
> 
> Is there some parameter or assumption that I'm not understanding, or did I 
> find a bug?

The quoting, if it were the correct quoting format, should be fine.   As to why 
it's the quote char and not the backtick, are you compiling these queries 
manually?You would want to make sure a MySQL dialect is in use, which would 
be using backticks for quoting, unless that dialect were initialized against a 
MySQL database that has ANSI_QUOTES set.

TL;DR quoting is a new thing here but SQLAlchemy should render the correct 
quotes when used with the MySQL dialect.

Here's a demo:

from sqlalchemy import Column
from sqlalchemy import create_engine
from sqlalchemy import Integer
from sqlalchemy import select
from sqlalchemy import String
from sqlalchemy.orm import declarative_base
Base = declarative_base()


class A(Base):
__tablename__ = 't_59a33cbea3617986d810e9fbae60ba19'

__table_args__ = {
"schema": "283ac7717fe770c5ed6d425c0c739cba"
}
id = Column(Integer, primary_key=True)
data = Column(String)

e = create_engine("mysql://")

stmt = select(A)

print(stmt.compile(e))

output:

SELECT 
`283ac7717fe770c5ed6d425c0c739cba`.t_59a33cbea3617986d810e9fbae60ba19.id, 
`283ac7717fe770c5ed6d425c0c739cba`.t_59a33cbea3617986d810e9fbae60ba19.data 
FROM `283ac7717fe770c5ed6d425c0c739cba`.t_59a33cbea3617986d810e9fbae60ba19



> 
> Thank you!
> Ben
> 
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/76d38390-e17f-4b90-a438-ee078944b5ffn%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/a50db81a-5271-4f3e-81c4-d4f02cc38732%40app.fastmail.com.

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

I really need to see:

1. actual models to the degree they illustrate the default loaders set up on 
relationships
2. query in use
3. **complete** stack trace, put it on a github gist if it's too long.  every 
line, the error, top to bottom.  thanks

On Fri, May 12, 2023, at 11:25 AM, Mike Bayer wrote:
> 
> 
> On Fri, May 12, 2023, at 1:25 AM, Shane Holcombe wrote:
>> AttributeError: 'NoneType' object has no attribute '_from_objects'
>> (I've shortened this from the entire traceback)
> 
> OK what kind of traceback, is it a recursion overflow kind of traceback ?  
> (e.g. many hundreds of repeated series of lines)? that would explain the 
> nonsensical error at least (there's no way for None to be in the place that 
> this error reports it)
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/9762bf53-8443-4780-9091-136863716290%40app.fastmail.com
>  
> <https://groups.google.com/d/msgid/sqlalchemy/9762bf53-8443-4780-9091-136863716290%40app.fastmail.com?utm_medium=email_source=footer>.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/b4f9ab0e-c402-4828-9db4-1385ad7b10a4%40app.fastmail.com.

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

On Fri, May 12, 2023, at 1:25 AM, Shane Holcombe wrote:
> AttributeError: 'NoneType' object has no attribute '_from_objects'
> (I've shortened this from the entire traceback)

OK what kind of traceback, is it a recursion overflow kind of traceback ?  
(e.g. many hundreds of repeated series of lines)? that would explain the 
nonsensical error at least (there's no way for None to be in the place that 
this error reports it)

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/9762bf53-8443-4780-9091-136863716290%40app.fastmail.com.

Re: [sqlalchemy] Migration issues - Column value None on select with joins

[Mike Bayer]

this has also been reported at 
https://github.com/sqlalchemy/sqlalchemy/discussions/9666



On Fri, May 12, 2023, at 1:25 AM, Shane Holcombe wrote:
> 
> From the digging around that I've done, it seems to be the ColumnLoader 
> strategy setup_query method sometimes has a value for 'c' of None where an 
> adapter is present but check_for_adapt is False. This is from the c = 
> adapter.columns[c] line.
> This happens only when called from the JoinedLoader class, it seems the 
> _generate_row_adapter method returns a clauses object with a columns dict 
> that has all the keys from the underlying table, but some of the values are 
> None.
> 
> All of this behaviour is fairly hard to get to happen however it seems to 
> occur after the warning "Loader depth for query is excessively deep; caching 
> will be disabled for additional loaders.  Consider using the recursion_depth 
> feature for deeply nested recursive eager loaders." occurs.
> This warning comes from a different part of the system where the base level 
> object, 'User', is very deeply nested, and on some later request to a 
> different part of the system, fetching a different 'User' relationship at a 
> less deep level triggers the above error.
> In our use case we have some fairly deeply nested relationships, 6 or so 
> levels deep, however we never used selectinload or immediateload to my 
> knowledge so there is no way to change the recursion depth.

we would need a reproduction case in order to see what this is about


> 
> Any help on this would be fantastic,
> Thanks,
> Shane
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/f6ee8641-0dc8-4eec-a79e-941ac3d814b9n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/a5f85728-b423-4eb3-9470-65bc12ccab6b%40app.fastmail.com.

Re: [sqlalchemy] Transaction integrity issue in SQLAlchemy SQLite dialect

[Mike Bayer]

the pysqlite driver does not deliver SERIALIZABLE isolation in its default mode 
of use.  you have to use workarounds to achieve this.  See the documentation at 
https://docs.sqlalchemy.org/en/20/dialects/sqlite.html#serializable-isolation-savepoints-transactional-ddl
 which details how to use these workarounds.  Also see 
https://github.com/python/cpython/issues/54133 
https://github.com/python/cpython/issues/83638 for related info.   pysqlite 
will have some new APIs for this in Python 3.12 but it's not clear to what 
extent they restore SERIALIZABLE isolation level without workarounds.



On Wed, May 10, 2023, at 6:58 PM, Roman Diba wrote:
> Summary:
> The SQLAlchemy SQLite dialect does not fully support serializable 
> transaction isolation in SQLite, leading to potential invalid data when 
> transactions interleave.
>
>
> Hello,
> This report documents an issue with transaction integrity in the 
> SQLAlchemy SQLite dialect.
> SQLite itself is able to fully support ACID transactions with 
> SERIALIZABLE isolation [1]. Also the Python sqlite3 library supports it 
> [2]. I'm demonstrating it in an example (roughly equivalent to my 
> SqlAlchemy code) [3].
>
> The situation in which I'm demonstrating the problem:
> 1) T1: Open transaction T1.
> 2) T1: Reads the entire `test` table.
> 3) T2: Open transaction T2.
> 4) T2: Add a row to the `test` table.
> 5) T2: Commit.
> 6) T1: Based on the read in step 2, calculate and insert a row into the 
> `test` table.
> 7) T1: Commit.
>
> The commit of T1 in step 7 should ROLLBACK because it does not reflect 
> the actual content of the table which was changed in step 5 by 
> transaction T2, and it would insert incorrect data.
>
> I have written code demonstrating this in SqlAlchemy [4].
>
> When run with a PostreSQL engine, it works as expected -- Transaction T1 
> rolls back:
> [...]
> INFO sqlalchemy.engine.Engine INSERT INTO test (value) VALUES 
> (%(value)s) RETURNING test.id
> INFO sqlalchemy.engine.Engine {'value': 'a0'}
> INFO sqlalchemy.engine.Engine COMMIT
> INFO sqlalchemy.engine.Engine INSERT INTO test (value) VALUES 
> (%(value)s) RETURNING test.id
> INFO sqlalchemy.engine.Engine {'value': 'b0'}
> INFO sqlalchemy.engine.Engine ROLLBACK
> [...]
> sqlalchemy.exc.OperationalError: 
> (psycopg2.extensions.TransactionRollbackError) could not serialize 
> access due to read/write dependencies among transactions
> DETAIL:  Reason code: Canceled on identification as a pivot, during write.
> HINT:  The transaction might succeed if retried.
>
>
> But when run with a SQLite engine, both transactions succeed:
> [...]
> INFO sqlalchemy.engine.Engine INSERT INTO test
> INFO sqlalchemy.engine.Engine ('a0',)
> INFO sqlalchemy.engine.Engine COMMIT
> INFO sqlalchemy.engine.Engine INSERT INTO test (value) VALUES (?)
> INFO sqlalchemy.engine.Engine ('b0',)
> INFO sqlalchemy.engine.Engine COMMIT
>
> After the final commit, there is invalid data in the `test` table.
>
> I expect this code should work similarly with both the PostgreSQL and 
> SQLite engines in SQLAlchemy.
>
>
> Tested in:
> - sqlalchemy 1.4.48
> - sqlalchemy 2.0.12
> - Python 3.7.10
>
> Thanks for your attention,
> Roman Diba, GLAMI
>
>
> [1] https://www.sqlite.org/isolation.html
> [2] https://docs.python.org/3/library/sqlite3.html#transaction-control
> [3] sqlite_isolation.py attached
> [4] sa_sqlite_isolation.py attached
>
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google 
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/8b61dd24-e893-5c8e-57d7-69616a272ec3%40glami.cz.
>
> Attachments:
> * sqlite_isolation.py
> * sa_sqlite_isolation.py

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/97af0dd4-e59f-4c69-a9b1-2bf89d246feb%40app.fastmail.com.

Re: [sqlalchemy] How to set Row instance attribute

[Mike Bayer]

On Sat, May 6, 2023, at 5:28 PM, sector119 wrote:
> Hello,
> 
> I get some data from DB and have a sequence of Row objects, how it is 
> possible to modify data in those objects attrs?

you can't.   copy them to something you can modify


> 
> Something like
> 
> result = await dbsession.execute(query)
> services = result.all()
> 
> for i, service in enumerate(services):
> services[i].title = f"some text here {service.title}"
> 
> but I get AttributeError("can't set attribute")
> 
> Thanks
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/a2552b95-e41b-4fa6-b655-3c16cf5f844en%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/9a38a417-cede-455e-bc64-5c4d7e73ace5%40app.fastmail.com.

Re: [sqlalchemy] DRYing up model relationships with custom properties

[Mike Bayer]

I'd be a little concerned about the double-underscore as it modifies the way 
Python getattr() works for those names, and I dont know that we have tests for 
that specific part of it, but besides that, you can always add new 
relationships or other mapped attributes to classes in one of two ways:

1. when using declarative Base, you can assign it:  ModelClass._field_for_rel = 
relationship()
2. you can set it in the mapper:  
ModelClass.__mapper__.add_property("_field_for_rel", relationship())

https://docs.sqlalchemy.org/en/20/orm/mapping_api.html#sqlalchemy.orm.Mapper.add_property

so you could make use of that code within some kind of function that is applied 
to the class with a certain name.

if you are looking to do it fully inline within the class, you might need to 
use event hooks that receive the class and mapper so that you can apply things 
after the fact, a good event for this would be "instrument_class": 
https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.MapperEvents.instrument_class
 

On Wed, May 3, 2023, at 4:18 AM, 'Tony Cosentini' via sqlalchemy wrote:
> Hi,
> 
> I have a pattern in some models where there is a private (as in prefixed with 
> __) relationship and then a property to handle some things that need to 
> happen when the relationship is fetched or written to.
> 
> Currently it's implemented like this:
> class ModelClass(Base):
>   __field_for_relationship = relationship('OtherModel')
>   @property
>   def field_for_relationship(self):
> # Some custom logic
> return self.__field_for_relationship
>  
>   @field_for_relationship.setter
>   def field_for_relationship(self, value: OtherModel):
> # Some custom logic.
> self.__field_for_relationship = value
> 
> This works, but I'd like to DRY this up into some kind of nice one-liner if 
> possible because I'm using this pattern on a handful of relationships. 
> However, it seems like the relationship field itself needs to be on the model 
> because of some magic that happens under the hood.
> 
> Any ideas on if it's possible to combine this into some kind of reusable 
> utility to DRY it up?
> 
> I'm not sure how important it is, but I'm still on 1.4.x, haven't made the 
> jump to 2.x yet.
> 
> Thanks!
> Tony
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/b285bdb1-c7b9-4734-9a5b-fe937241b44an%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/1eef8072-96b9-4883-87ab-3587a28f2c38%40app.fastmail.com.

Re: [sqlalchemy] Exceeding the queue pool limit in a Flask application problem

[Mike Bayer]

are you making sure you create only one global OrmEngine object for the whole 
application?  

SQLA docs recommend engines are global

On Fri, Apr 28, 2023, at 12:48 PM, Erabil Erabil wrote:
> When using SQL Alchemy in a Flask application, if the application 
> continuously sends queries to the database, it exceeds the maximum number of 
> connections in the connection pool. How can I solve this issue?
> 
> *sqlalchemy_engine.py*
> 
> **
> 
> import os
> from sqlalchemy import create_engine
> from sqlalchemy.orm import sessionmaker
> from sqlalchemy.pool import QueuePool
> 
> 
> class OrmEngine:
> def __init__(self):
> self.engine = create_engine(os.getenv('DATABASE_URL'),
> poolclass=QueuePool,
> pool_size=5,
> max_overflow=3
> )
> self.session = sessionmaker(bind=self.engine)
> *Example query*
> 
> with self.orm_engine.session() as session:
> user_repository = UserRepository(session)
> *user = user_repository.find_by_email(login_request.email)** *
> 
> *My connection list*
> 
> **
> 
> MariaDB [(none)]> SHOW PROCESSLIST;
> +-+-+-+--+-+--+--+--+--+
> | Id  | User| Host| db   | Command | Time | State 
>| Info | Progress |
> +-+-+-+--+-+--+--+--+--+
> |   2 | system user | | NULL | Daemon  | NULL | InnoDB purge 
> worker  | NULL |0.000 |
> |   1 | system user | | NULL | Daemon  | NULL | InnoDB purge 
> coordinator | NULL |0.000 |
> |   3 | system user | | NULL | Daemon  | NULL | InnoDB purge 
> worker  | NULL |0.000 |
> |   4 | system user | | NULL | Daemon  | NULL | InnoDB purge 
> worker  | NULL |0.000 |
> |   5 | system user | | NULL | Daemon  | NULL | InnoDB 
> shutdown handler  | NULL |0.000 |
> |   8 | root| localhost:56603 | NULL | Query   |0 | Init  
>| SHOW PROCESSLIST |0.000 |
> | 645 | root| localhost:64281 | tbot | Sleep   |   74 |   
>| NULL |0.000 |
> | 662 | root| localhost:64334 | tbot | Sleep   |   56 |   
>| NULL |0.000 |
> | 684 | root| localhost:64400 | tbot | Sleep   |   33 |   
>| NULL |0.000 |
> | 705 | root| localhost:64476 | tbot | Sleep   |   16 |   
>| NULL |0.000 |
> | 707 | root| localhost:64482 | tbot | Sleep   |   14 |   
>| NULL |0.000 |
> | 709 | root| localhost:64488 | tbot | Sleep   |   13 |   
>| NULL |0.000 |
> | 711 | root| localhost:64494 | tbot | Sleep   |   12 |   
>| NULL |0.000 |
> | 713 | root| localhost:64500 | tbot | Sleep   |   10 |   
>| NULL |0.000 |
> | 715 | root| localhost:64506 | tbot | Sleep   |9 |   
>| NULL |0.000 |
> | 717 | root| localhost:64512 | tbot | Sleep   |8 |   
>| NULL |0.000 |
> | 719 | root| localhost:64518 | tbot | Sleep   |7 |   
>| NULL |0.000 |
> | 720 | root| localhost:64521 | tbot | Sleep   |4 |   
>| NULL |0.000 |
> | 722 | root| localhost:64527 | tbot | Sleep   |3 |   
>| NULL |0.000 |
> | 724 | root| localhost:64533 | tbot | Sleep   |2 |   
>| NULL |0.000 |
> | 726 | root| localhost:64539 | tbot | Sleep   |1 |   
>| NULL |0.000 |
> +-+-+-+--+-+--+--+--+--+
> 21 rows in set (0.000 sec)
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> 

Re: [sqlalchemy] StaleDataError on simple update statement

[Mike Bayer]

this can happen if you are using the wrong datatype for your primary key.  like 
it's a string in the database and you are referring to it as an Integer, that 
kind of thing.

no way to provide more detail without seeing a short version of the mapping 
that produces this error and CREATE TABLE statement used in the database.

On Wed, Apr 26, 2023, at 6:23 AM, Mirel Glejser wrote:
> I am completely stumped after working on this issue for 2 days.
> 
> I am simply querying a user, updating the name, and sending it back to the 
> database.
> 
> user = session.query(User).get(1)
> user.first_name # John 
> user.first_name = 'Sally' 
> session.commit()
> 
> # > sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'user' 
> expected to update 1 row(s); # -1 were matched.
> 
> What is causing this error?
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/0eed0704-5c66-48f4-a41f-73512b18f858n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/176253d9-5dcf-4484-934d-b07741113003%40app.fastmail.com.

Re: [sqlalchemy] Modifying Query Object

[Mike Bayer]

if you add a column to a Query that was returning an object, like Data, you 
will get back tuples of (Data, extra_col).  this is probably what the tool you 
are using is complaining about.

On Fri, Apr 14, 2023, at 10:51 PM, Mike Bayer wrote:
> 
> 
> On Tue, Apr 11, 2023, at 9:17 PM, Luis Del Rio IV wrote:
>> The sql query itself returns several rows, as it should. But when trying to 
>> combine the max using sqlalchemy the rows return as the following.
>> 
>> Received incompatible instance \"(> 0x7f2c6cfd6670>, '2021-04-10 18', Decimal('7294.0'))\".",
> 
> that's not a SQLAlchemy error.   I would again suggest you get help from the 
> maintainers of the library you're using.
> 
> 
> 
>> 
>> Here I am able to get the max for that row group, but am unable to pass it 
>> into the DataModel object
>> 
>> On Tuesday, April 11, 2023 at 4:59:15 PM UTC-7 S Mahabl wrote:
>>> Do you get many rows?
>>> 
>>> SELECT  date_format(data.timestamp, "%Y-%m-%d %H"), max(data.value)  AS 
>>> data_value
>>> from data
>>> GROUP BY date_format(data.timestamp, "%Y-%m-%d %H")
>>> On Tue, Apr 11, 2023 at 4:24 PM Luis Del Rio IV  wrote:
>>>> Hello,
>>>> 
>>>> I am currently using the following sqlalchemy code,
>>>> 
>>>> _query = super().get_query(model, info, sort, **args)
>>>> query = _query.group_by(
>>>> func.date_format(DataModel.timestamp, "%Y-%m-%d %H")
>>>> )
>>>> return query
>>>> 
>>>> I am trying to aggregate the the max value of a field from DataModel.value 
>>>> utilizing the group by clause.
>>>> 
>>>> In simple sql, you would do the following.
>>>> 
>>>> SELECT  max(data.value)  AS data_value
>>>> from data
>>>> GROUP BY date_format(data.timestamp, "%Y-%m-%d %H")
>>>> 
>>>> What would the proper way to express this? The various methods I have 
>>>> tried somehow overwrite the original query and do not map to our 
>>>> attributes.
>>>> 
>>>> 
>>>> --
>>>> SQLAlchemy -
>>>> The Python SQL Toolkit and Object Relational Mapper
>>>>  
>>>> http://www.sqlalchemy.org/
>>>>  
>>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>>> description.
>>>> ---
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "sqlalchemy" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to sqlalchemy+...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/sqlalchemy/a73c1830-8c64-437a-8ea7-a171767e2223n%40googlegroups.com
>>>>  
>>>> <https://groups.google.com/d/msgid/sqlalchemy/a73c1830-8c64-437a-8ea7-a171767e2223n%40googlegroups.com?utm_medium=email_source=footer>.
>> 
>> 
>> --
>> SQLAlchemy -
>> The Python SQL Toolkit and Object Relational Mapper
>>  
>> http://www.sqlalchemy.org/
>>  
>> To post example code, please provide an MCVE: Minimal, Complete, and 
>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>> description.
>> ---
>> You received this message because you are subscribed to the Google Groups 
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to sqlalchemy+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/sqlalchemy/3eeda16b-9c36-4ca5-91b5-cba325343fban%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/sqlalchemy/3eeda16b-9c36-4ca5-91b5-cba325343fban%40googlegroups.com?utm_medium=email_source=footer>.
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy

Re: [sqlalchemy] Modifying Query Object

[Mike Bayer]

On Tue, Apr 11, 2023, at 9:17 PM, Luis Del Rio IV wrote:
> The sql query itself returns several rows, as it should. But when trying to 
> combine the max using sqlalchemy the rows return as the following.
> 
> Received incompatible instance \"( 0x7f2c6cfd6670>, '2021-04-10 18', Decimal('7294.0'))\".",

that's not a SQLAlchemy error.   I would again suggest you get help from the 
maintainers of the library you're using.



> 
> Here I am able to get the max for that row group, but am unable to pass it 
> into the DataModel object
> 
> On Tuesday, April 11, 2023 at 4:59:15 PM UTC-7 S Mahabl wrote:
>> Do you get many rows?
>> 
>> SELECT  date_format(data.timestamp, "%Y-%m-%d %H"), max(data.value)  AS 
>> data_value
>> from data
>> GROUP BY date_format(data.timestamp, "%Y-%m-%d %H")
>> On Tue, Apr 11, 2023 at 4:24 PM Luis Del Rio IV  wrote:
>>> Hello,
>>> 
>>> I am currently using the following sqlalchemy code,
>>> 
>>> _query = super().get_query(model, info, sort, **args)
>>> query = _query.group_by(
>>> func.date_format(DataModel.timestamp, "%Y-%m-%d %H")
>>> )
>>> return query
>>> 
>>> I am trying to aggregate the the max value of a field from DataModel.value 
>>> utilizing the group by clause.
>>> 
>>> In simple sql, you would do the following.
>>> 
>>> SELECT  max(data.value)  AS data_value
>>> from data
>>> GROUP BY date_format(data.timestamp, "%Y-%m-%d %H")
>>> 
>>> What would the proper way to express this? The various methods I have tried 
>>> somehow overwrite the original query and do not map to our attributes.
>>> 
>>> 
>>> -- 
>>> SQLAlchemy - 
>>> The Python SQL Toolkit and Object Relational Mapper
>>>  
>>> http://www.sqlalchemy.org/
>>>  
>>> To post example code, please provide an MCVE: Minimal, Complete, and 
>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
>>> description.
>>> --- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to sqlalchemy+...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/sqlalchemy/a73c1830-8c64-437a-8ea7-a171767e2223n%40googlegroups.com
>>>  
>>> .
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/3eeda16b-9c36-4ca5-91b5-cba325343fban%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/ee62e370-b0c1-4656-95e2-4ecbcf4791e7%40app.fastmail.com.

Re: [sqlalchemy] Re: Dealing with readonly column

[Mike Bayer]

On Fri, Apr 14, 2023, at 3:02 PM, Lele Gaifax wrote:
> "Mike Bayer"  writes:
>
>> On Fri, Apr 14, 2023, at 8:03 AM, Lele Gaifax wrote:
>>> I now have
>>>
>>>   CREATE TABLE something (id SERIAL, name TEXT)
>>>
>>>   CREATE FUNCTION counter(something)
>>>   RETURNS INTEGER AS $$
>>> SELECT count(*) FROM something_else se
>>> WHERE se.something_id = $1.id
>>>   $$ STABLE SQL
>>>
>>> and thus existing queries such as
>>>
>>>   SELECT s.name, s.counter
>>>   FROM something AS s
>>>
>>> work as before.
>>
>> how does "SELECT s.counter" work if the table does not have an actual 
>> "counter" column?  
>
> As explained here [1], "the field notation and functional notation are
> equivalent", that is "SELECT a.foo FROM t AS a" and "SELECT foo(a) FROM
> t AS a" means the same, when "foo" is a function accepting the "implicit
> composite type corresponding to a row in the table t".

oh geez it's that silly PostgreSQL syntax.not a fan.   you can use that but 
you'd be on your own

>
>>
>> Using Computed in this way is equivalent to using FetchedValue.Why is 
>> that a problem? 
>
> No, it's not a problem, but the following
>
>   stmt = something.update().values(counter=42).where(something.c.id=1)
>   connection.execute(stmt)
>  
> will raise an error at execution time, as will, assuming
> "instance_of_something" is an instance of the ORM class mapped to the
> table "something"

So what then is a "read only column" if not one that raises an error if someone 
tries to write to it explicitly?

>   instance_of_something.counter = 42
>   session.flush()
>
> I was just imagining something that could raise an error earlier.

well you have the ORM validates and the before_cursor_execute approaches, but I 
would think if this is simple developer level programming guards, the PG 
exception is perfect


>
> Unfortunately in this case PG does not help: the closest thing is a
> "GENERATED" column [2], but (up to version 15 at least) it "is a special
> column that is always computed from other columns", it cannot be an
> arbitrary subquery.

trigger, then.  im sure this is a thing PG can do

>
>> Without using server-side constructs, in a practical sense, simply
>> omitting the column from the Table or using mapper.exclude_properties
>> is in my experience sufficient.
>
> Uhm, how could I then select that value, to be returned by an API that
> queries the "something" table?

OK so the "readonly" you are looking for includes one where your application 
actually needs to load it, then fine, dont exclude it from your mappings.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/eb9fc1e0-8494-4ec8-aad5-4cab3ae10bf5%40app.fastmail.com.

Re: [sqlalchemy] Using joins+max with sql server

[Mike Bayer]

this line of code:

query = session.query(CodigoProduto.CdChamada, Produto.IdProduto, 
Produto.NmProduto, ProdutoEmpresa.VlPrecoCusto, ProdutoEmpresa.VlPrecoSugerido, 
EstoqueEmpresa.DtReferencia, EstoqueEmpresa.QtEstoque)


should look like this:

query = session.query(CodigoProduto.CdChamada, Produto.IdProduto, 
Produto.NmProduto, ProdutoEmpresa.VlPrecoCusto, ProdutoEmpresa.VlPrecoSugerido, 
estoqueAtual.DtReferencia, estoqueAtual.QtEstoque)




On Fri, Apr 14, 2023, at 3:21 PM, Elias Coutinho wrote:
> *I am suffering!
> 
> It showed the same message.*
> 
> 
> # Subquery para buscar o maior registro de estoqueempresa para cada produto
> estoqueAtual = session.query(
> EstoqueEmpresa.IdProduto,
> EstoqueEmpresa.QtEstoque,
> func.max(EstoqueEmpresa.DtReferencia).label('MaxDtReferencia')
> ).group_by(EstoqueEmpresa.IdProduto, EstoqueEmpresa.QtEstoque).subquery()
> 
> #print(estoqueAtual)
> 
> # Realiza a consulta com SQLAlchemy
> ##query = session.query(CodigoProduto.CdChamada, Produto.IdProduto, 
> Produto.NmProduto, ProdutoEmpresa.VlPrecoCusto, 
> ProdutoEmpresa.VlPrecoSugerido, estoqueAtual.c.QtEstoque)\
> ##   .join(ProdutoEmpresa, Produto.IdProduto == 
> ProdutoEmpresa.IdProduto)\
> ##   .join(CodigoProduto, Produto.IdProduto == 
> CodigoProduto.IdProduto)\
> ##   .join(estoqueAtual, and_(Produto.IdProduto == 
> estoqueAtual.c.IdProduto))\
> ##   .join(EstoqueEmpresa, and_(EstoqueEmpresa.IdProduto == 
> Produto.IdProduto))\
> ##   .filter(ProdutoEmpresa.StAtivoVenda == 'S')\
> ##   .filter(ProdutoEmpresa.CdEmpresa == 4)\
> ##   .order_by(Produto.NmProduto)
> 
> query = session.query(CodigoProduto.CdChamada, Produto.IdProduto, 
> Produto.NmProduto, ProdutoEmpresa.VlPrecoCusto, 
> ProdutoEmpresa.VlPrecoSugerido, EstoqueEmpresa.DtReferencia, 
> EstoqueEmpresa.QtEstoque)\
>.join(ProdutoEmpresa, Produto.IdProduto == 
> ProdutoEmpresa.IdProduto)\
>.join(CodigoProduto, Produto.IdProduto == 
> CodigoProduto.IdProduto)\
>.join(estoqueAtual, and_(Produto.IdProduto == 
> estoqueAtual.c.IdProduto, EstoqueEmpresa.DtReferencia == 
> estoqueAtual.c.MaxDtReferencia))\
>.join(EstoqueEmpresa, and_(EstoqueEmpresa.IdProduto == 
> Produto.IdProduto, EstoqueEmpresa.DtReferencia == 
> estoqueAtual.c.MaxDtReferencia))\
>.filter(ProdutoEmpresa.StAtivoVenda == 'S')\
>    .filter(ProdutoEmpresa.CdEmpresa == 4)\
>.order_by(Produto.NmProduto)
> 
> Em sex., 14 de abr. de 2023 às 15:21, Mike Bayer 
>  escreveu:
>> __
>> the initial issue is that you want DtReferencia from the subquery on the 
>> outside:
>> 
>> session.query(..., estoqueAtual.c.DtReferencia, ...)
>> 
>> and not "EstoqueEmpresa.DtReferencia", that's not available in the FROM 
>> list, it's inside a subquery.
>> 
>> also I dont think you'd want to "group by" the same column that you are 
>> feeing into max().that would defeat the purpose of using an aggregate.
>> 
>> On Fri, Apr 14, 2023, at 1:30 PM, Elias Coutinho wrote:
>>> Good afternoon.
>>> I am having trouble transforming a SQL Server query to SQL Alchemy.
>>> 
>>> *The SQL Server query is this*
>>> **
>>> SELECT CP.CdChamada, P.NmProduto, PE.VlPrecoCusto, PE.VlPrecoSugerido, 
>>> EE.QtEstoque, EE.DtReferencia
>>> FROM Produto P
>>>  INNER JOIN Produto_Empresa PE ON  P.IdProduto = PE.IdProduto
>>>  INNER JOIN CodigoProduto CP ON  P.IdProduto = CP.IdProduto
>>>  INNER JOIN (SELECT IdProduto, CdEmpresa, MAX(DtReferencia) AS 
>>> MaxDtReferencia
>>>  FROM EstoqueEmpresa
>>>  GROUP BY IdProduto, CdEmpresa) AS EE2 ON EE2.IdProduto = 
>>> P.IdProduto AND EE2.CdEmpresa = PE.CdEmpresa
>>>  INNER JOIN EstoqueEmpresa EE ON EE.IdProduto = EE2.IdProduto AND 
>>> EE.DtReferencia = EE2.MaxDtReferencia AND EE.CdEmpresa = EE2.CdEmpresa
>>> WHERE PE.StAtivoVenda = 'S' AND
>>>   PE.CdEmpresa = 4 AND
>>>   CP.IdTipoCodigoProduto = '00A002'
>>> ORDER BY CP.CdChamada**
>>> 
>>> *My code is as follows:*
>>> *My Model*
>>> class EstoqueEmpresa(Base):
>>> __tablename__ = 'EstoqueEmpresa'
>>> 
>>> IdProduto = Column(CHAR(10, 'SQL_Latin1_General_CP850_CI_AI'), 
>>> primary_key=True, nullable=False)
>>> CdEmpresa = Column(Integer, primary_key=True, nullable=False)
>>> DtReferencia = Column(DateTime, primary_key=True, nullable=False, 
>>> i

Re: [sqlalchemy] Using joins+max with sql server

[Mike Bayer]

the initial issue is that you want DtReferencia from the subquery on the 
outside:

session.query(..., estoqueAtual.c.DtReferencia, ...)

and not "EstoqueEmpresa.DtReferencia", that's not available in the FROM list, 
it's inside a subquery.

also I dont think you'd want to "group by" the same column that you are feeing 
into max().that would defeat the purpose of using an aggregate.

On Fri, Apr 14, 2023, at 1:30 PM, Elias Coutinho wrote:
> Good afternoon.
> I am having trouble transforming a SQL Server query to SQL Alchemy.
> 
> *The SQL Server query is this*
> **
> SELECT CP.CdChamada, P.NmProduto, PE.VlPrecoCusto, PE.VlPrecoSugerido, 
> EE.QtEstoque, EE.DtReferencia
> FROM Produto P
>  INNER JOIN Produto_Empresa PE ON  P.IdProduto = PE.IdProduto
>  INNER JOIN CodigoProduto CP ON  P.IdProduto = CP.IdProduto
>  INNER JOIN (SELECT IdProduto, CdEmpresa, MAX(DtReferencia) AS MaxDtReferencia
>  FROM EstoqueEmpresa
>  GROUP BY IdProduto, CdEmpresa) AS EE2 ON EE2.IdProduto = 
> P.IdProduto AND EE2.CdEmpresa = PE.CdEmpresa
>  INNER JOIN EstoqueEmpresa EE ON EE.IdProduto = EE2.IdProduto AND 
> EE.DtReferencia = EE2.MaxDtReferencia AND EE.CdEmpresa = EE2.CdEmpresa
> WHERE PE.StAtivoVenda = 'S' AND 
>   PE.CdEmpresa = 4 AND 
>   CP.IdTipoCodigoProduto = '00A002'
> ORDER BY CP.CdChamada**
> 
> *My code is as follows:*
> *My Model*
> class EstoqueEmpresa(Base):
> __tablename__ = 'EstoqueEmpresa'
> 
> IdProduto = Column(CHAR(10, 'SQL_Latin1_General_CP850_CI_AI'), 
> primary_key=True, nullable=False)
> CdEmpresa = Column(Integer, primary_key=True, nullable=False)
> DtReferencia = Column(DateTime, primary_key=True, nullable=False, 
> index=True)
> VersaoRegistro = Column(TIMESTAMP, nullable=False)
> QtCompra = Column(Float(53))
> VlCompra = Column(Float(53))
> QtVenda = Column(Float(53))
> VlVenda = Column(Float(53))
> VlCustoMercadoriaVendida = Column(Float(53))
> QtEntrada = Column(Float(53))
> VlEntrada = Column(Float(53))
> QtSaida = Column(Float(53))
> VlSaida = Column(Float(53))
> VlSaidaAcerto = Column(Float(53))
> QtSaidaAcerto = Column(Float(53))
> QtEstoque = Column(Float(53))
> VlEstoque = Column(Float(53))
> VlUltimoCustoMedio = Column(Float(53))
> DtInicialAlinhamentoEstoque = Column(DateTime)
> QtCompraNaoAtualizaCustoMedio = Column(Float(53))
> VlCompraNaoAtualizaCustoMedio = Column(Float(53))
> QtEntradaNaoAtualizaCustoMedio = Column(Float(53))
> VlEntradaNaoAtualizaCustoMedio = Column(Float(53))
> **
> *My code*
> from sqlalchemy import create_engine, text, Column, update, insert, select
> from sqlalchemy.orm.exc import NoResultFound
> from sqlalchemy.orm import Session
> from sqlalchemy.orm import sessionmaker
> from sqlalchemy import func, and_
> 
> import unidecode
> import pandas as pd
> from datetime import datetime
> import re
> import itertools
> 
> from itertools import islice
> 
> # Importo somente as tabelas que vou usar
> from models import CodigoProduto, ProdutoEmpresa, EstoqueEmpresa, Produto
> 
> #-
> # Configuração da conexão com o banco de dados
> USER = 'sa'
> PASSWORD = 'Abc*123'
> HOST = 'SERVER-02\MSSQLSERVERB'
> DATABASE = 'ALTERDATA_TESTE'
> 
> engine = 
> create_engine(f'mssql+pyodbc://{USER}:{PASSWORD}@{HOST}/{DATABASE}?driver=ODBC+Driver+17+for+SQL+Server')
> 
> # Create a Session object
> Session = sessionmaker(bind=engine)
> session = Session()
> 
> 
> #-
> # Subquery para buscar o maior registro de estoqueempresa para cada produto
> estoqueAtual = session.query(
> EstoqueEmpresa.IdProduto,
> EstoqueEmpresa.QtEstoque,
> func.max(EstoqueEmpresa.DtReferencia).label('MaxDtReferencia')
> ).group_by(EstoqueEmpresa.IdProduto, EstoqueEmpresa.QtEstoque, 
> EstoqueEmpresa.DtReferencia).subquery()
> 
> #print(estoqueAtual)
> 
> # Realiza a consulta com SQLAlchemy
> ##query = session.query(CodigoProduto.CdChamada, Produto.IdProduto, 
> Produto.NmProduto, ProdutoEmpresa.VlPrecoCusto, 
> ProdutoEmpresa.VlPrecoSugerido, estoqueAtual.c.QtEstoque)\
> ##   .join(ProdutoEmpresa, Produto.IdProduto == 
> ProdutoEmpresa.IdProduto)\
> ##   .join(CodigoProduto, Produto.IdProduto == 
> CodigoProduto.IdProduto)\
> ##   .join(estoqueAtual, and_(Produto.IdProduto == 
> estoqueAtual.c.IdProduto))\
> ##   .join(EstoqueEmpresa, and_(EstoqueEmpresa.IdProduto == 
> Produto.IdProduto))\
> ##   .filter(ProdutoEmpresa.StAtivoVenda == 'S')\
> ##   .filter(ProdutoEmpresa.CdEmpresa == 4)\
> ##   .order_by(Produto.NmProduto)
> 
> query = session.query(CodigoProduto.CdChamada, Produto.IdProduto, 
> Produto.NmProduto, 

Re: [sqlalchemy] Dealing with readonly column

[Mike Bayer]

On Fri, Apr 14, 2023, at 8:03 AM, Lele Gaifax wrote:
> Hi,
>
> I wonder if there is a way to declare a particular column of a table as
> "readonly", either for the purpose of documenting the model, or to get
> early error should someone try to update it.

"readonly" can mean a few different things

>
> Implementing a new "feature", where I have to rewrite the logic that
> keeps up-to-date a "counter" column, I thought of replacing that column
> with a "computed value" (a.k.a. "virtual column"), using a PostgreSQL
> SQL function that compute that counter on-the-fly [1]: this allowed me
> to avoid touching dozens of places where that column is read in the PHP
> code (what a relief!).
>
> To illustrate, where I had a table such as
>
>   CREATE TABLE something (id SERIAL, name TEXT, counter INTEGER)
>
> I now have
>
>   CREATE TABLE something (id SERIAL, name TEXT)
>
>   CREATE FUNCTION counter(something)
>   RETURNS INTEGER AS $$
> SELECT count(*) FROM something_else se
> WHERE se.something_id = $1.id
>   $$ STABLE SQL
>
> and thus existing queries such as
>
>   SELECT s.name, s.counter
>   FROM something AS s
>
> work as before.

how does "SELECT s.counter" work if the table does not have an actual "counter" 
column?  


>
> For now, I just left the SA Table definition (I'm using classic mappings)
> untouched, that is something like
>
>   from sqlalchemy import Column, Integer, MetaData, Table, Text
>
>   metadata = MetaData()
>
>   something = Table("something", metadata,
> Column("id", Integer, primary_key=True),
> Column("name", Text),
> Column("counter", Integer))
>
> but as said, I looked around to see if there is a way to inform SA that
> that "counter" column cannot be updated, just read. I tried to use the
> Computed() thing like

so here's where "readonly" means different things.  if you want to omit 
"counter" from your application entirely, you can remove it from the Table() 
definition.

Also with some irony I am actually working on a whole different feature right 
now where there's a new parameter, which I'm not going to make public yet, 
called "_omit_from_statements" for an individual column.From an ORM mapping 
perspective, it's essentially the same as if you added this column to the 
Mapper exclude_properties list: 
https://docs.sqlalchemy.org/en/20/orm/mapping_api.html#sqlalchemy.orm.Mapper.params.exclude_properties
 which will also make that column not present within the mapping or any of the 
statements used.

neither of these two things make the column "read only", they just make the 
column not really present in the application.

In general, whenever a column has a server side default of this nature, you 
want to use FetchedValue or some subclass of this in the column definition, so 
the ORM as well as Core knows that something in the server will be generating a 
value for this column: 
https://docs.sqlalchemy.org/en/20/core/defaults.html#triggered-columns / 
https://docs.sqlalchemy.org/en/20/orm/persistence_techniques.html#orm-server-defaults
 

>
>   from sqlalchemy import Computed
>  
>   something = Table("something", metadata,
> Column("id", Integer, primary_key=True),
> Column("name", Text),
> Column("counter", Integer, Computed("counter")))
>
> but accordingly to the documentation [2] "SA behavior ... is currently
> that the value [assigned to the column] will be ignored", so that does
> not bring me any advantage.

Using Computed in this way is equivalent to using FetchedValue.Why is that 
a problem? 


So far note all of these techniques have to do with "read only", but they are 
the usual approaches taken when there's a server generated column that you'd 
prefer the application not deal with explicitly.

>
> On the ORM side, I could "hide" the concrete column and expose it thru a
> readonly property, but I'd like to have a similar safety-belt also at
> the Core layer, because many services/APIs will be written at that
> level.
>
> Do you have any suggestion?

true "readonly" at the ORM level, meaning the column is present, it's mapped, 
and you want to disallow user code from saying myclass.col = "foo" or some 
error is raised, you can only do with attribute events or with a validator: 
https://docs.sqlalchemy.org/en/20/orm/mapped_attributes.html#sqlalchemy.orm.validates
 or other Python-side object technique.

true "readonly" at the Core level, where any attempt to use 
connection.execute() in such a way that data is being sent to this column, 
you'd need to use event hooks such as before_cursor_execute() 
https://docs.sqlalchemy.org/en/20/core/events.html#sqlalchemy.events.ConnectionEvents.before_cursor_execute
 and then do heuristics to figure out what the statement is, and if any of the 
parameters point to this column.I would not go this route since it's 
overkill and not that reliable for said overkill.

If you want to 

Re: [sqlalchemy] Query object modification

[Mike Bayer]

On Thu, Apr 13, 2023, at 4:14 PM, Luis Del Rio IV wrote:
> Mike,
> 
> Would this select query be able to get our aggregated data?
> 
> query = 
> select(func.max(DataModel.value)).select_from(_query.subquery()).group_by(
> func.date_format(DataModel.timestamp, "%Y-%m-%d %H")
> )
> 
> We tried this route but are now getting this error
> 
>   "errors": [
> {
>   "message": "'Select' object has no attribute 'statement'",
>   "locations": [
> {
>   "line": 2,
>   "column": 3
> }
>   ],

While there is no stack trace that would illustrate the line of code that is 
actually making this mistake and this is not illustrated in the line of code 
indicated above, it looks like you are attempting to call upon an attribute 
named .statement, which is something that's on Query, but not on select(), so 
if you have a select() object, you would not want to call .statement first.


-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/6a8a3b62-3340-4da0-839d-5c606ef4d9a7%40app.fastmail.com.

Re: [sqlalchemy] Query object modification

[Mike Bayer]

On Thu, Apr 13, 2023, at 1:13 PM, Peter Harrison wrote:
> Thanks Mike,
> 
> Ideally we'd prefer to find a solution via Graphene-SQLAlchemy. Unfortunately 
> we don't have the luxury of creating our own query when interacting with 
> Graphene-SQLAlchemy.
> 
> So the key question for us is, can you modify an existing 
> sqlalchemy.orm.query.Query object to insert a "func.max(Data.value)" object 
> into the select?

yes, use the add_columns method

https://docs.sqlalchemy.org/en/20/orm/queryguide/query.html#sqlalchemy.orm.Query.add_columns


> 
> 
> 
> This is what Graphene-SQLAlchemy is giving us.
> If this is possible, then the group_by part is easy. We have tested that 
> frequently, the hard part is the modifying the original "select" object.
> 
> 1. We have tried add_columns, but that adds in incompatible object type in 
> the GraphQL results making it an unusable option.

I dont know what that means... wouldn't this be an issue for 
graphene-SQLAlchemy to fix?  this is a basic Query method.  if you can query 
individual columns to GraphQL, it should work.  Then as stated before, if 
whatever graphene-sqlalchemy does is not compatible with individual columns, 
that would have to be resolved on that end.


> 2. We thought that modifying the select would be  possible using data with 
> "statement.froms" but can't figure out how to update the MetaData object in it

again this is very vague and doesn't make much sense.   if you want to add a 
fromclause to a select() or a query, there's an add_froms() method.   This has 
nothing to do with updating MetaData objects, which are not part of the "froms" 
of a select and are only a collection that Table objects belong towards, and 
they have nothing to do with how SELECT statements are rendered.

> 
> If modifying the "select" after its creation is not possible, we need to 
> start considering using a separate reporting table with hourly data.

yes unfortunately I know nothing about graphene-sqlalchemy or graphql and these 
sound like details that are deeply embedded in a larger system, the people who 
created this extension would have the best chance of helping with new 
integrations.   

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/275f600c-86fd-4870-bf83-79699d4f7164%40app.fastmail.com.

Re: [sqlalchemy] Access other columns in a custom type

[Mike Bayer]

On Thu, Apr 13, 2023, at 6:10 AM, Hussein Samadi wrote:
> Hi everyone.
> I'm creating a new SQLA custom type using TypeDecorator base class. Is it 
> possible to have access to the value of other fields in 
> "process_result_value" method? Generally, is it possible to create a custom 
> type that have access to other column's data in the de-serialization process?

not at all, you would need to approach your problem at a different level.


> 
> Thank you
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/4a61f34d-1e47-4294-884f-4fb03570698bn%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/48012641-ccb6-454e-979d-3801ff135891%40app.fastmail.com.

Re: [sqlalchemy] Query object modification

[Mike Bayer]

On Wed, Apr 12, 2023, at 5:21 PM, Luis Del Rio IV wrote:
> I am currently using the following sqlalchemy code,
> 
> _query = super().get_query(model, info, sort, **args)
> query = _query.group_by(
> func.date_format(DataModel.timestamp, "%Y-%m-%d %H")
> )
> return query
> 
> I am trying to aggregate the the max value of a field from DataModel.value 
> utilizing the group by clause.
> 
> In simple sql, you would do the following.
> 
> SELECT  max(data.value)  AS data_value
> from data
> GROUP BY date_format(data.timestamp, "%Y-%m-%d %H")
> 
> What would the proper way to express this? The various methods I have tried 
> somehow overwrite the original query and do not map to our attributes.

using legacy query style:

q1 = session.query(func.max(Data.value)).group_by(
func.date_format(Data.timestamp, "%Y-%m-%d %H")
)


using 2.0 style select(), replace "session.query" with "select":

s1 = select(func.max(Data.value)).group_by(
func.date_format(Data.timestamp, "%Y-%m-%d %H")
)

POC script is at the bottom of this email.

> 
> 
> I have some additional context on the issue
>  1. We are using the graphene-sqlalchemy package.
>  2. When you do a GraphQL web api query, the package creates a 
> sqlalchemy.orm.query.Query object.
>  3. We want to modify this standard query that the package creates so that we 
> can do the group_by action to help us get the maximum time series value for 
> every hour because this is not possible using GraphQL.
>  4. Talking to the graphene-sqlalchemy team they told us the best place to do 
> the modification to the standardized query is in the get_query method (line 
> 67) in this file: 
> https://github.com/graphql-python/graphene-sqlalchemy/blob/master/graphene_sqlalchemy/fields.py
>  5. The standardized query we need to modify translates to this SQL 
> statement: 
> SELECT sy_data.oid_id, sy_data.rrd_timestamp, sy_data.cabinet_id, sy_data
> .customer_id, sy_data.value, sy_data.active_flag, sy_data.timestamp 
> FROM sy_data 
> WHERE sy_data.rrd_timestamp >= %(rrd_timestamp_1)s AND sy_data.rrd_timestamp 
> <= %(rrd_timestamp_2)s AND (sy_data.oid_id = %(oid_id_1)s OR sy_data.oid_id = 
> %(o
> id_id_2)s) ORDER BY sy_data.oid_id ASC, sy_data.rrd_timestamp ASC
> 
> Therefore we need to find a way to insert a func.max for the values in the 
> first part of the SELECT statement, (before the FROM). It is easy for us to 
> apend the group_by like this.

if you have a Query which renders the above SQL, you can add group_by() to it 
in place.   but if these queries are being translated into GraphQL, and GraphQL 
does not have any concept of GROUP BY, then it wont work, what you want to do 
would not be possible unless a GraphQL query exists that does what you need.

> 
> query.group_by(
> func.date_format(DataModel.timestamp, "%Y-%m-%d %H")
> )
> 
> The big issue for us is to figure out how to insert the func.max

see func at 
https://docs.sqlalchemy.org/en/20/core/sqlelement.html#sqlalchemy.sql.expression.func


from sqlalchemy import Column
from sqlalchemy import DateTime
from sqlalchemy import func
from sqlalchemy import Integer
from sqlalchemy import select
from sqlalchemy.orm import declarative_base
from sqlalchemy.orm import Session

Base = declarative_base()


class Data(Base):
__tablename__ = "data"

id = Column(Integer, primary_key=True)
value = Column(Integer)
timestamp = Column(DateTime)


s = Session()

q1 = s.query(func.max(Data.value)).group_by(
func.date_format(Data.timestamp, "%Y-%m-%d %H")
)

print(q1)

s1 = select(func.max(Data.value)).group_by(
func.date_format(Data.timestamp, "%Y-%m-%d %H")
)

print(s1)

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/328c16af-189a-4050-a67f-06c78a9dec99%40app.fastmail.com.

Re: [sqlalchemy] missing schema authorization clause

[Mike Bayer]

I've never heard of this concept before so a google into stack overflow shows 
https://stackoverflow.com/questions/10994414/missing-authorization-clause-while-creating-schema,
 where you are probably looking for:

with engine.begin() as conn:
conn.execute(text("create user {schema_name} identified by {password}"))



On Wed, Apr 5, 2023, at 6:26 PM, sumau wrote:
> Hello
> 
> Is it possible to pass in the authorization when creating a schema with the 
> oracle dialect?
> 
> oracle_engine = create_engine(
> f"oracle://{user}:{password}@{host}:{port}/{database_name}"
> )
> inspector = inspect(oracle_engine)
> if schema_name not in inspector.get_schema_names():
> oracle_engine.execute(CreateSchema(schema_name))
> 
> I tried to pass in the authorization as a kw variable but no luck:
> 
> if schema not in inspector.get_schema_names():
> connectable.execute(CreateSchema(schema), authorization=user)
> 
> Regards
> Soumaya
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/ff2f5a13-c17c-43ea-a469-041a306a2e62n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/88deed52-d70f-4905-8832-03efca819237%40app.fastmail.com.

Re: [sqlalchemy] relationships between inheritance tables

[Mike Bayer]

this is not so much about the database models as it is about the queries being 
emitted and what kinds of data are being returned.

I would go through the section at 
https://docs.sqlalchemy.org/en/20/faq/performance.html#how-can-i-profile-a-sqlalchemy-powered-application
 and identify where the issue exists.

On Thu, Mar 30, 2023, at 1:14 AM, Kaan Baha Sever wrote:
> Hello! I have three post type and inherit from post table. And this post have 
> comment and like table foreign key. But I am using this method my performance 
> very very low. One page loading 3-4 sec. How can I do?
> My Tables
> Post(Model):
> id
> name
> 
> PostType1(Post):
> id
> content
> 
> PostType2(Post):
> id
> body
> PostTyp1.id foreign key
> 
> PostType3(Post):
> id
> body
> PostTyp2.id foreign key
> 
> Comment(Model)
> Post.id
> 
> Like(Model)
> Post.id and I want to comment ID
> 
> 
> 
> 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/3bfb5d25-b017-473d-9e19-73d5e723d323n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/af4c5429-9032-47e9-96c8-3fa0d1a804f9%40app.fastmail.com.

Re: [sqlalchemy] TLS 1.2

[Mike Bayer]

this is a driver issue so you'd need to get specifics from the driver you're 
using, such as mysqlclient or pymysql. 

per https://dev.mysql.com/doc/c-api/8.0/en/c-api-encrypted-connections.html it 
looks like "TLS_VERSION"

create_engine("mysql+mysqldb://...?ssl=true", connect_args={"ssl": 
{"tls_version": "1.2"}})

background on customizing connections are at 
https://docs.sqlalchemy.org/en/20/core/engines.html#use-the-connect-args-dictionary-parameter
 



On Mon, Mar 27, 2023, at 2:39 PM, Patrick Bruce wrote:
> I am connecting to a MySQL db using sqlalchemy and TLS 1.2 is enforced server 
> side. I connect by setting my URI to the correct connection string and adding 
> the ?ssl=true flag at the end. However I am getting an error that the TLS 
> version does not meet the minimum requirements of the server. My question is 
> how to force sqlalchemy to connect using TLS 1.2?
> 
> App env: Python 3.6 using Flask 
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/8803031f-ca72-454a-bd73-ad3b0bc65773n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/4b1cf0a4-99d7-48a5-b06c-fc12d864f08f%40app.fastmail.com.

Re: [sqlalchemy] How to use sqlcodegen generate SQLModels

[Mike Bayer]

hey there-

sqlacodegen support is on their github at 
https://github.com/agronholm/sqlacodegen/discussions/categories/q-a



On Wed, Mar 22, 2023, at 6:55 PM, Linson Abah wrote:
> Hi,
> I am trying to convert the schema of a db to SQLModels: I have this short 
> script:
> 
> *from subprocess import call
> outfile = 'models/dvdrental_models.py'
> generator = 'sqlmodels'
> call(['sqlacodegen', db_uri, '--generator', generator, '--outfile', outfile])
*
> I got this Error.
> sqlacodegen: error: unrecognized arguments: --generator sqlmodels
> 
> How do I modify this to work correctly? 
> 
> Thank you
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/ecc2b9f3-73a5-4a9a-b02a-4495d22779e2n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/6e146e19-e1a5-4f21-a78c-dd89c4a042fc%40app.fastmail.com.

Re: [sqlalchemy] Test query seems to spuriously give sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown column 'tb_br.id' in 'on clause'")

[Mike Bayer]

MySQL's "only full group by" option causes MySQL to use the SQL standard for 
GROUP BY, which states that all columns that are not contained within an 
aggregate function must be in the GROUP BY clause.

That is, this SQL is invalid:

SELECT table.x, max(table.y) FROM table

The reason is that max(table.y) is going to SELECT only one row, the row that 
has the highest value for "y".  Where does that leave ".x" , does it just 
indicate the value of "x" that just happens to be at the max(y) row?  What if 
there are 100 "y" rows all with that max value?From a SQL standpoint, the 
function is wrong.

Instead, the correct way to write the above SQL is:

SELECT table.x, max(table.y) FROM table GROUP BY table.x

Above, the max() function is now applied to individual groups of rows, and we 
get multiple rows back, one row for each value of "x" along with the max(y) for 
that group.

The MySQL error you are getting is basically asking you to add "NV.id" to the 
GROUP BY clause.

Background on this concept is at 
https://dev.mysql.com/doc/refman/5.7/en/group-by-handling.html .

it's critical though that you understand the SQL you want to emit and confirm 
it gets back the result you are looking for.   Just adding elements to the 
query until MySQL accepts it is not necessarily going to get you the results 
you want.


On Tue, Mar 21, 2023, at 7:04 PM, 'Dan Stromberg [External]' via sqlalchemy 
wrote:
> Granted, it’s difficult to read my (admittedly rather blank) mind.  Maybe I 
> should restate the question:
>  
> What are my options?  I just want to see the rows from the query below.  Why 
> is it telling me I need to aggregate, and if I do truly need to, what might 
> be an aggregate function that won’t eliminate much of what the query is 
> producing?
>  
> *From: *'Dan Stromberg [External]' via sqlalchemy 
> 
> *Date: *Tuesday, March 21, 2023 at 9:05 AM
> *To: *sqlalchemy@googlegroups.com 
> *Subject: *Re: [sqlalchemy] Test query seems to spuriously give 
> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown 
> column 'tb_br.id' in 'on clause'")
> Alright, using join_from may have led to clearing a hurdle. I’m now using: 
> query = ( select(NV. id, func. min(bs_3. build_id)) .select_from(bs) 
> .join(v_2, onclause=(bs. version_id == v_2. id)) .join_from(bs_2, Br, 
> onclause=(Br. id == bs_2. branch_id))
> 
>  
> Alright, using join_from may have led to clearing a hurdle.  I’m now using:
> query = (
> select(NV.id, func.min(bs_3.build_id))
> .select_from(bs)
> .join(v_2, onclause=(bs.version_id == v_2.id))
> .join_from(bs_2, Br, onclause=(Br.id == bs_2.branch_id))
> )
>  
> ..and am getting a new error:
> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1140, "In 
> aggregated query without GROUP BY, expression #1 of SELECT list contains 
> nonaggregated column 'tact_dev.tb_nv.id'; this is incompatible with 
> sql_mode=only_full_group_by")
>  
> Do I need to aggregate?  Or perhaps change sql_mode?
>  
> Thanks!
>  
> *From: *sqlalchemy@googlegroups.com  on behalf 
> of Mike Bayer 
> *Date: *Monday, March 20, 2023 at 5:33 PM
> *To: *noreply-spamdigest via sqlalchemy 
> *Subject: *Re: [sqlalchemy] Test query seems to spuriously give 
> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown 
> column 'tb_br.id' in 'on clause'")
> OK, not really, you want tables in the FROM clause. use either the 
> select_from() or join_from() method to do that: https: //docs. sqlalchemy. 
> org/en/20/tutorial/data_select. html#explicit-from-clauses-and-joins On Mon, 
> Mar 20, 2023, at 5: 16 PM,
> OK, not really, you want tables in the FROM clause.  use either the 
> select_from() or join_from() method to do that:
>  
> https://docs.sqlalchemy.org/en/20/tutorial/data_select.html#explicit-from-clauses-and-joins
>  
> <https://urldefense.com/v3/__https:/docs.sqlalchemy.org/en/20/tutorial/data_select.html*explicit-from-clauses-and-joins__;Iw!!Ci6f514n9QsL8ck!gsSZYRslnIShc80D5SJP9hQv7FJkNL5Bzfvc8dkqobmEg8-ctkAcRyR1sZuv3pRL4eCzLvlJC-VDSf5sXXQNtX0d4POMpzTQh3-QUw$>
>  
>  
>  
> On Mon, Mar 20, 2023, at 5:16 PM, 'Dan Stromberg [External]' via sqlalchemy 
> wrote:
>>  
>> 
>> I’m getting some pushback internally, from my team lead – he and I both 
>> think it’s probably too much detail to share.  It’s 43 lines of SQL with 
>> multiple subqueries.
>> 
>>  
>> 
>> Would just the simplest parts of the from clause work?
>> 
>>  
>> 
>> *From: *sqlalchemy@googlegroups.com  on behalf 
>> of Mike Bayer 
>> *Date: *Monday, March 20, 2023 at 1:11 PM
>> *To: *noreply-s

Re: [sqlalchemy] Working with an existing database, reflection and usage of ORM Declarative Mapping

[Mike Bayer]

ill clarify that para in the reflection part

On Tue, Mar 21, 2023, at 11:15 AM, Pierre Massé wrote:
> Thanks for this quick answer Mike!
> 
> Obivously, I wasted way to much energy in trying to make everything work with 
> table reflection... :(  Let's say I gained experience from it.
> 
> Regarding the origin of the impression I might have gotten, I'd say I tried 
> to read a big chunk of the docs (unified tutorial and ORM docs) prior to 
> moving to ORM, and got some nudges towards this understanding:
> - I do not think that I read something like "__tablename__ and attribute 
> names must be the same like in the underlying database" anywhere (but was not 
> thorough in my reading of the docs). I will try to see what exception I will 
> get if I try to map a non-existing field :)
> - In the unified tutorial 
> <https://docs.sqlalchemy.org/en/20/tutorial/metadata.html#table-reflection>, 
> I misunderstood this part: "*Table reflection refers to the process of 
> generating Table 
> <https://docs.sqlalchemy.org/en/20/core/metadata.html#sqlalchemy.schema.Table>
>  and related objects by reading the current state of a database. Whereas in 
> the previous sections we’ve been declaring Table 
> <https://docs.sqlalchemy.org/en/20/core/metadata.html#sqlalchemy.schema.Table>**
>  objects in Python and then emitting DDL to the database, the reflection 
> process does it in reverse.*" like if it was one of 2 choices
> - And the overall quality of the other docs entries that described so well 
> how to "post declare" attributes, how to defer reflection, etc... that 
> enabled a beginner to pull of such a complex application
> 
> Not sure if this misconception is widespread, but maybe some tweaking of the 
> docs may have prevented this error. I'd be happy to contribute, but have 
> never pull requested to an open source repo.
> 
> Anyway, thanks for the answer, wish you a pleasant day!
> 
> Regards,
> 
> Pierre
> Le mardi 21 mars 2023 à 13:54:59 UTC+1, Mike Bayer a écrit :
>> __
>> hi -
>> 
>> I think things would be easier if you defined your ORM mappings/ tables 
>> without relying upon reflection.   There is no such requirement that 
>> reflection is used for an existing database, you just want to have ORM table 
>> metadata that matches the schema.   the ORM/table metadata does not have to 
>> match the schema exactly either, it can omit columns and tables you aren't 
>> using.
>> 
>> reflection is more of a utility function that can be used for some special 
>> cases but for a first class app /database combination it would not be 
>> prevalent. It's a bit curious where you might have gotten the impression 
>> that "reflection is mandatory when working with an existing database".
>> 
>> On Tue, Mar 21, 2023, at 7:58 AM, Pierre Massé wrote:
>>> Dear all,
>>> 
>>> I am quite new to ORMs and SQLAlchemy, and I have a maybe somewhat naive 
>>> question set regarding how to build an application around a database.
>>> 
>>> Just a few words of context (may not yield importance, but who knows): I am 
>>> building a mobile app, for which the server side will be an AWS Lambda 
>>> function serving GraphQL queries. Data persistence is achieved through a 
>>> hosted PostgreSQL instance. Server side code is python and "database 
>>> access" through SQLAlchemy. The first version of the database schema has 
>>> been "manually" built and populated with some test data (via simple SQL 
>>> queries in pgAdmin4).
>>> 
>>> Regarding SQLAlchemy usage, first version was using Core only, but I 
>>> decided to move to ORM, and I got it quite hard - maybe because of poor 
>>> choices on my end.
>>> 
>>> What I do, now that I have a working example:
>>> - when the Lambda is fired, I import a module defining "bare" ORM classes, 
>>> with no attribute apart the table name - inheriting from *`Reflected`* and 
>>> a declarative base (using deferred reflection 
>>> <https://docs.sqlalchemy.org/en/20/orm/declarative_tables.html#using-deferredreflection>)
>>> - engine is set up and connection established to the db
>>> - *`Reflected`* class is prepared through *Reflected.prepare(engine=engine)*
>>> - New attributes, relationships, etc... are added to the "bare" ORM classes 
>>> as *`column_property`*, after reflection (this way 
>>> <https://docs.sqlalchemy.org/en/20/orm/mapped_sql_expr.html#adding-column-property-to-an-existing-declarative-mapped-class>)
>>> - Mapper is then ready, da

Re: [sqlalchemy] Working with an existing database, reflection and usage of ORM Declarative Mapping

[Mike Bayer]

hi -

I think things would be easier if you defined your ORM mappings/ tables without 
relying upon reflection.   There is no such requirement that reflection is used 
for an existing database, you just want to have ORM table metadata that matches 
the schema.   the ORM/table metadata does not have to match the schema exactly 
either, it can omit columns and tables you aren't using.

reflection is more of a utility function that can be used for some special 
cases but for a first class app /database combination it would not be 
prevalent. It's a bit curious where you might have gotten the impression 
that "reflection is mandatory when working with an existing database".

On Tue, Mar 21, 2023, at 7:58 AM, Pierre Massé wrote:
> Dear all,
> 
> I am quite new to ORMs and SQLAlchemy, and I have a maybe somewhat naive 
> question set regarding how to build an application around a database.
> 
> Just a few words of context (may not yield importance, but who knows): I am 
> building a mobile app, for which the server side will be an AWS Lambda 
> function serving GraphQL queries. Data persistence is achieved through a 
> hosted PostgreSQL instance. Server side code is python and "database access" 
> through SQLAlchemy. The first version of the database schema has been 
> "manually" built and populated with some test data (via simple SQL queries in 
> pgAdmin4).
> 
> Regarding SQLAlchemy usage, first version was using Core only, but I decided 
> to move to ORM, and I got it quite hard - maybe because of poor choices on my 
> end.
> 
> What I do, now that I have a working example:
> - when the Lambda is fired, I import a module defining "bare" ORM classes, 
> with no attribute apart the table name - inheriting from *`Reflected`* and a 
> declarative base (using deferred reflection 
> )
> - engine is set up and connection established to the db
> - *`Reflected`* class is prepared through *Reflected.prepare(engine=engine)*
> - New attributes, relationships, etc... are added to the "bare" ORM classes 
> as *`column_property`*, after reflection (this way 
> )
> - Mapper is then ready, data is queried and mutated using sessions
> 
> My questions are:
> - is reflection mandatory when working with an existing database? (I think 
> this would be like an overwhelmingly prevalent case for Production 
> applications?)
> - is it possible to have a mixed approach regarding the mapping definitions: 
> some attributes being defined in the ORM classes prior to the reflection, and 
> reflection then completes those classes with other fields from the database 
> schema?
> - when using reflection, is the only way to define new attributes, 
> relationships, etc... to add those attributes after this reflection via 
> adding column_properties after class definition, like described above?
> - I feel like I am losing much of the "Declarative Mapping" by working the 
> way I do, what do you think about it?
> - overall, what could be simplified regarding the ways of working I set up?
> 
> Some code snippets below:
> 
> _Bare ORM class definitions:_
> class MessageOrm(Reflected, Base):
> __tablename__ = "single_recipient_message"
> 
> 
> class HeaderOrm(Reflected, Base):
> __tablename__ = "single_recipient_message_header"
> __
> _Post reflection addition of relationships and attributes_
> HeaderOrm.messages = relationship(
> MessageOrm,
> foreign_keys=[MessageOrm.header_id],
> back_populates="header",
> )
> MessageOrm.sent_by_who = column_property(
> case(
> (MessageOrm.sender_id == current_id, "me"),
> else_="other",
> )
> )
> MessageOrm.header = relationship(
> "HeaderOrm",
> foreign_keys=[MessageOrm.header_id],
> back_populates="messages",
> )
> __
> Thanks a lot!
> 
> Regards,
> 
> Pierre
> __
> __
> __
> __
> __
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/433f9917-7ac0-4d1b-b41e-16d8ae255d15n%40googlegroups.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve 

Re: [sqlalchemy] Test query seems to spuriously give sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown column 'tb_br.id' in 'on clause'")

[Mike Bayer]

OK, not really, you want tables in the FROM clause.  use either the 
select_from() or join_from() method to do that:

https://docs.sqlalchemy.org/en/20/tutorial/data_select.html#explicit-from-clauses-and-joins



On Mon, Mar 20, 2023, at 5:16 PM, 'Dan Stromberg [External]' via sqlalchemy 
wrote:
>  
> I’m getting some pushback internally, from my team lead – he and I both think 
> it’s probably too much detail to share.  It’s 43 lines of SQL with multiple 
> subqueries.
>  
> Would just the simplest parts of the from clause work?
>  
> *From: *sqlalchemy@googlegroups.com  on behalf 
> of Mike Bayer 
> *Date: *Monday, March 20, 2023 at 1:11 PM
> *To: *noreply-spamdigest via sqlalchemy 
> *Subject: *Re: [sqlalchemy] Test query seems to spuriously give 
> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown 
> column 'tb_br.id' in 'on clause'")
> what SQL are you going for ? start with that. On Mon, Mar 20, 2023, at 10: 33 
> AM, 'Dan Stromberg [External]' via sqlalchemy wrote: That makes sense, but…. 
> I’m afraid I don’t know how to add tb_br to the select. I tried: query = ( 
> select(NV. id,
> 
> what SQL are you going for ?  start with that.
>  
> On Mon, Mar 20, 2023, at 10:33 AM, 'Dan Stromberg [External]' via sqlalchemy 
> wrote:
>> That makes sense, but…. I’m afraid I don’t know how to add tb_br to the 
>> select.
>> 
>>  
>> 
>> I tried:
>> 
>> query = (
>> 
>> select(NV.id, func.min(bs_3.build_id))
>> 
>> .select_from(bs, Br)
>> 
>> .join(v_2, onclause=(bs.version_id == v_2.id))
>> 
>> .join(bs_2, onclause=(Br.id == bs_2.branch_id))
>> 
>> )
>> 
>> …which gave:
>> 
>> 1054, "Unknown column 'tb_br.id' in 'on clause'"
>> 
>>  
>> 
>>  
>> 
>> …and I tried:
>> query = (
>> select(NV.id, func.min(bs_3.build_id), Br)
>> 
>> .select_from(bs)
>> 
>> .join(v_2, onclause=(bs.version_id == v_2.id))
>> 
>> .join(bs_2, onclause=(Br.id == bs_2.branch_id))
>> 
>> )
>> 
>> …which also gave:
>> 
>> (1054, "Unknown column 'tb_br.id' in 'on clause'")
>> 
>>  
>> 
>> I’m guessing I’m missing something simple, but I have no idea what.
>> 
>>  
>> 
>> Any (further) suggestions?
>> 
>>  
>> 
>>  
>> 
>> *From: *sqlalchemy@googlegroups.com  on behalf 
>> of Mike Bayer 
>> *Date: *Saturday, March 18, 2023 at 8:01 AM
>> *To: *noreply-spamdigest via sqlalchemy 
>> *Subject: *Re: [sqlalchemy] Test query seems to spuriously give 
>> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown 
>> column 'tb_br.id' in 'on clause'")
>> 
>> the query emitted is: SELECT tb_nv. id, min(bs_3. build_id) AS min_1 FROM 
>> tb_nv, tb_brst AS bs_3, tb_brst AS bs INNER JOIN tb_vers AS v_2 ON bs. 
>> version_id = v_2. id INNER JOIN tb_brst AS bs_2 ON tb_br. id = bs_2. 
>> branch_id the error means that your
>> 
>>  
>> 
>> the query emitted is:
>> 
>>  
>> 
>> SELECT tb_nv.id, min(bs_3.build_id) AS min_1
>> 
>> FROM tb_nv, tb_brst AS bs_3, tb_brst AS bs
>> 
>> INNER JOIN tb_vers AS v_2 ON bs.version_id = v_2.id
>> 
>> INNER JOIN tb_brst AS bs_2 ON tb_br.id = bs_2.branch_id
>> 
>>  
>> 
>> the error means that your ON clause refers to a table "tb_br" which is not 
>> otherwise in the FROM clause: "ON tb_br.id = bs_2.branch_id"
>> 
>>  
>> 
>> the ON clause can only refer to columns from tables that are being SELECTed 
>> from, such as:
>> 
>>  
>> 
>> SELECT tb_nv.id, min(bs_3.build_id) AS min_1
>> 
>> FROM tb_nv, tb_brst AS bs_3, tb_brst AS bs
>> 
>> INNER JOIN tb_vers AS v_2 ON bs.version_id = v_2.id
>> 
>> INNER JOIN tb_br ON tb_br.id = bs_2.branch_id
>> 
>> INNER JOIN tb_brst AS bs_2 ON tb_br.id = bs_2.branch_id
>> 
>>  
>> 
>> so you'd need to alter your query to include some indication how tb_br is 
>> part of what's being joined.
>> 
>>  
>> 
>> On Fri, Mar 17, 2023, at 7:52 PM, 'Dan Stromberg' via sqlalchemy wrote:
>> 
>>>  
>>> 
>>> Hi people.
>>> 
>>>  
>>> 
>>> I'm having trouble with a test query.
>>> 
>>>  
>>> 
>>> As the subject line says, I'm getting:
>>> 
>>> sqlalchemy.exc.OperationalError: (MySQLd

Re: [sqlalchemy] Test query seems to spuriously give sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown column 'tb_br.id' in 'on clause'")

[Mike Bayer]

what SQL are you going for ?  start with that.

On Mon, Mar 20, 2023, at 10:33 AM, 'Dan Stromberg [External]' via sqlalchemy 
wrote:
> That makes sense, but…. I’m afraid I don’t know how to add tb_br to the 
> select.
>  
> I tried:
> query = (
> select(NV.id, func.min(bs_3.build_id))
> .select_from(bs, Br)
> .join(v_2, onclause=(bs.version_id == v_2.id))
> .join(bs_2, onclause=(Br.id == bs_2.branch_id))
> )
> …which gave:
> 1054, "Unknown column 'tb_br.id' in 'on clause'"
>  
>  
> …and I tried:
> query = (
> 
> select(NV.id, func.min(bs_3.build_id), Br)
> .select_from(bs)
> .join(v_2, onclause=(bs.version_id == v_2.id))
> .join(bs_2, onclause=(Br.id == bs_2.branch_id))
> )
> …which also gave:
> (1054, "Unknown column 'tb_br.id' in 'on clause'")
>  
> I’m guessing I’m missing something simple, but I have no idea what.
>  
> Any (further) suggestions?
>  
>  
> *From: *sqlalchemy@googlegroups.com  on behalf 
> of Mike Bayer 
> *Date: *Saturday, March 18, 2023 at 8:01 AM
> *To: *noreply-spamdigest via sqlalchemy 
> *Subject: *Re: [sqlalchemy] Test query seems to spuriously give 
> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown 
> column 'tb_br.id' in 'on clause'")
> the query emitted is: SELECT tb_nv. id, min(bs_3. build_id) AS min_1 FROM 
> tb_nv, tb_brst AS bs_3, tb_brst AS bs INNER JOIN tb_vers AS v_2 ON bs. 
> version_id = v_2. id INNER JOIN tb_brst AS bs_2 ON tb_br. id = bs_2. 
> branch_id the error means that your
> 
> the query emitted is:
>  
> SELECT tb_nv.id, min(bs_3.build_id) AS min_1
> FROM tb_nv, tb_brst AS bs_3, tb_brst AS bs
> INNER JOIN tb_vers AS v_2 ON bs.version_id = v_2.id
> INNER JOIN tb_brst AS bs_2 ON tb_br.id = bs_2.branch_id
>  
> the error means that your ON clause refers to a table "tb_br" which is not 
> otherwise in the FROM clause: "ON tb_br.id = bs_2.branch_id"
>  
> the ON clause can only refer to columns from tables that are being SELECTed 
> from, such as:
>  
> SELECT tb_nv.id, min(bs_3.build_id) AS min_1
> FROM tb_nv, tb_brst AS bs_3, tb_brst AS bs
> INNER JOIN tb_vers AS v_2 ON bs.version_id = v_2.id
> INNER JOIN tb_br ON tb_br.id = bs_2.branch_id
> INNER JOIN tb_brst AS bs_2 ON tb_br.id = bs_2.branch_id
>  
> so you'd need to alter your query to include some indication how tb_br is 
> part of what's being joined.
>  
> On Fri, Mar 17, 2023, at 7:52 PM, 'Dan Stromberg' via sqlalchemy wrote:
>>  
>> Hi people.
>>  
>> I'm having trouble with a test query.
>>  
>> As the subject line says, I'm getting:
>> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown 
>> column 'tb_br.id' in 'on clause'")
>>  
>> But it seems like tb_br exists, and has an id column - tb_br being an empty 
>> table, but still, existent:
>> mysql> show create table tb_br;
>> +---+-+
>> | Table | Create Table   
>>  
>> |
>> +---+-+
>> | tb_br | CREATE TABLE `tb_br` (
>>   `id` int(11) NOT NULL AUTO_INCREMENT,
>>   `name` varchar(45) NOT NULL,
>>   PRIMARY KEY (`id`)
>> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
>> +---+-+
>> 1 row in set (0.04 sec)
>>  
>> mysql> select * from tb_br;
>> Empty set (0.03 sec)
>>  
>> The query, along with sample models, looks like:
>> #!/usr/bin/env python3
>>  
>> """
>> A little test program.
>>  
>> Environment variables:
>> DBU  Your database user
>> DBP  Your database password
>> DBH  Your database host
>> IDB  Your initial database
>> """
>>  
>> import os
>> import pprint
>>  
>> from sqlalchemy import create_engine, select
>> from sqlalchemy.orm import aliased, sessionmaker, declarative_base
>> from sqlalchemy.sql.expression import func
>> from flask_sqlalchemy import SQLAlchemy
>>  
>> db = SQLAlchemy()
&g

Re: [sqlalchemy] Test query seems to spuriously give sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown column 'tb_br.id' in 'on clause'")

[Mike Bayer]

the query emitted is:

SELECT tb_nv.id, min(bs_3.build_id) AS min_1
FROM tb_nv, tb_brst AS bs_3, tb_brst AS bs
INNER JOIN tb_vers AS v_2 ON bs.version_id = v_2.id
INNER JOIN tb_brst AS bs_2 ON tb_br.id = bs_2.branch_id

the error means that your ON clause refers to a table "tb_br" which is not 
otherwise in the FROM clause: "ON tb_br.id = bs_2.branch_id"

the ON clause can only refer to columns from tables that are being SELECTed 
from, such as:

SELECT tb_nv.id, min(bs_3.build_id) AS min_1
FROM tb_nv, tb_brst AS bs_3, tb_brst AS bs
INNER JOIN tb_vers AS v_2 ON bs.version_id = v_2.id
INNER JOIN tb_br ON tb_br.id = bs_2.branch_id
INNER JOIN tb_brst AS bs_2 ON tb_br.id = bs_2.branch_id

so you'd need to alter your query to include some indication how tb_br is part 
of what's being joined.

On Fri, Mar 17, 2023, at 7:52 PM, 'Dan Stromberg' via sqlalchemy wrote:
> 
> Hi people.
> 
> I'm having trouble with a test query.
> 
> As the subject line says, I'm getting:
> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1054, "Unknown 
> column 'tb_br.id' in 'on clause'")
> 
> But it seems like tb_br exists, and has an id column - tb_br being an empty 
> table, but still, existent:
> mysql> show create table tb_br;
> +---+-+
> | Table | Create Table
>   
>   |
> +---+-+
> | tb_br | CREATE TABLE `tb_br` (
>   `id` int(11) NOT NULL AUTO_INCREMENT,
>   `name` varchar(45) NOT NULL,
>   PRIMARY KEY (`id`)
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
> +---+-+
> 1 row in set (0.04 sec)
> 
> mysql> select * from tb_br;
> Empty set (0.03 sec)
> 
> The query, along with sample models, looks like:
> #!/usr/bin/env python3
> 
> """
> A little test program.
> 
> Environment variables:
> DBU  Your database user
> DBP  Your database password
> DBH  Your database host
> IDB  Your initial database
> """
> 
> import os
> import pprint
> 
> from sqlalchemy import create_engine, select
> from sqlalchemy.orm import aliased, sessionmaker, declarative_base
> from sqlalchemy.sql.expression import func
> from flask_sqlalchemy import SQLAlchemy
> 
> db = SQLAlchemy()
> Base = declarative_base()
> 
> 
> class NV(Base):
> __tablename__ = "tb_nv"
> __bind_key__ = "testdb"
> __table_args__ = (
> {
> "mysql_engine": "InnoDB",
> "mysql_charset": "utf8",
> "mysql_collate": "utf8_general_ci",
> },
> )
> 
> id = db.Column("id", db.Integer, primary_key=True, autoincrement=True)
> builds = db.relationship("Bld", primaryjoin="(NV.id == Bld.variant_id)")
> 
> 
> class Vers(Base):
> __tablename__ = "tb_vers"
> __bind_key__ = "testdb"
> __table_args__ = (
> {
> "mysql_engine": "InnoDB",
> "mysql_charset": "utf8",
> "mysql_collate": "utf8_general_ci",
> },
> )
> 
> id = db.Column("id", db.Integer, primary_key=True, autoincrement=True)
> 
> 
> class St(Base):
> __tablename__ = "tb_brst"
> __bind_key__ = "testdb"
> __table_args__ = ({"mysql_engine": "InnoDB", "mysql_charset": "utf8"},)
> 
> id = db.Column("id", db.Integer, primary_key=True, autoincrement=True)
> version_id = db.Column(
> "version_id",
> db.Integer,
> db.ForeignKey(
> "tb_vers.id",
> name="fk_tb_brst_version_id",
> onupdate="CASCADE",
> ondelete="RESTRICT",
> ),
> nullable=False,
> )
> branch_id = db.Column(
> "branch_id",
> db.Integer,
> db.ForeignKey(
> "tb_br.id",
> name="fk_tb_brst_branch_id",
> onupdate="CASCADE",
> ondelete="RESTRICT",
> ),
> nullable=False,
> )
> build_id = db.Column(
> "build_id",
> db.Integer,
> db.ForeignKey(
> "tb_bld.id",
> name="fk_tb_brst_build_id",
> onupdate="CASCADE",
> ondelete="RESTRICT",
> ),
> nullable=False,
> )
> 
> version = db.relationship(
> "Vers", innerjoin=True, primaryjoin="(St.version_id == Vers.id)"
> )
> branch = db.relationship(
> "Br", innerjoin=True, primaryjoin="(St.branch_id == Br.id)"
> )
> build = db.relationship(
> "Bld", innerjoin=True, primaryjoin="(St.build_id == Bld.id)"
> )
> 
> 
> class Br(Base):
> __tablename__ = 

Re: [sqlalchemy] Using connection pools with individual PostgreSQL user logins for a backend REST API

[Mike Bayer]

I think you should use a single master login for create_engine() so that the 
pool deals with connections only at that top role.

Then when you **use** connections, use PostgreSQL SET ROLE: 
https://www.postgresql.org/docs/current/sql-set-role.html to set the current 
role on the connection / session you are using, and RESET ROLE to turn it back 
before connections are returned to the pool.

When you start a request, set the role:

await session.connection().exec_driver_sql("SET ROLE current_user")

Then, assuming you are using SET ROLE and not SET LOCAL ROLE, you want to reset 
this role upon return to the pool.  Establish a pool return handler that calls 
RESET ROLE using the "checkin" event: 
https://docs.sqlalchemy.org/en/20/core/events.html#sqlalchemy.events.PoolEvents.checkin
 

@event.listens_for(engine, "checkin")
def receive_checkin(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("RESET ROLE")
cursor.close()



hope this helps






On Sun, Feb 26, 2023, at 11:10 PM, Nate J wrote:
> Hi List,
> 
> I have no Google-fu. This is especially obvious when trying to search for 
> something like I’m asking about.
> 
> Does anyone have any info or links to docs, blog posts, etc. that covers how 
> to use connection pools with individual PostgreSQL user logins for a backend 
> REST API?
> 
> The reason I ask is I’m using row level security (RLS) like in the example 
> below from the PostgreSQL docs. I’ve only ever used a single 
> application-specific account for accessing the database for all tenants. Now, 
> however, each API key used in an API request will be tied to an account table 
> and will have a PostgreSQL database user created for them so “current_user” 
> works for RLS.
> 
> I’d like for requests to complete as fast as possible and I’m guessing 
> there’s a better way than logging in to the database for each request.
> 
> The software involved is PostgreSQL 14, SQLAlchemy with asyncpg and FastAPI.
> 
> From the page: https://www.postgresql.org/docs/current/ddl-rowsecurity.html
> 
> One example uses the DDL and policy:
> 
> CREATE TABLE accounts (manager text, company text, contact_email text);
> 
> ALTER TABLE accounts ENABLE ROW LEVEL SECURITY;
> 
> CREATE POLICY account_managers ON accounts TO managers
> USING (manager = current_user);
> 
> Thanks for your help.
> 
> 
> -- 
> SQLAlchemy - 
> The Python SQL Toolkit and Object Relational Mapper
>  
> http://www.sqlalchemy.org/
>  
> To post example code, please provide an MCVE: Minimal, Complete, and 
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> description.
> --- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sqlalchemy/DCF5536D-71FD-46BA-B831-0755588A79DD%40gmail.com
>  
> .

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/b0a3a566-4e69-42d0-a24a-d8139dff950d%40app.fastmail.com.