serem53 commented on issue #67546:
URL: https://github.com/apache/doris/issues/67546#issuecomment-7047456879
A second, separate symptom of the C extension in dbt-doris, which also goes
away with `use_pure=True` / `threads: 1`: **a segfault at startup when
`threads` > 1.**
**Environment**
- dbt-core 1.12.2, dbt-doris 1.0.0, mysql-connector-python 9.6.0 (`HAVE_CEXT
= True`)
- Python 3.13.14, Debian 12 (bookworm), x86_64, run from an Airflow worker
container
- Apache Doris 4.0.3-rc03
**What happens**
With `threads: 2` in the profile, `dbt build` sometimes dies about 20 s in,
before any model runs. There's no Python traceback and no dbt error. stdout
stops after `Concurrency: 2 threads`, stderr has only:
```
resource_tracker.py: UserWarning: resource_tracker: There appear to be 2
leaked semaphore objects to clean up at shutdown
```
`logs/dbt.log` always ends at the same point, with two threads opening their
first connection at the same moment:
```
[ThreadPool]: Acquiring new doris connection 'list_'
[ThreadPool]: Acquiring new doris connection 'list_'
[ThreadPool]: Using doris connection "list_"
[ThreadPool]: Using doris connection "list_"
[ThreadPool]: Opening a new connection, currently in state init
[ThreadPool]: Opening a new connection, currently in state init
<log ends>
```
This hit 5 of 14 scheduled builds over three days. On `threads: 1` we had
about 200 builds with the same versions and the same server, and none crashed
this way.
**Repro outside dbt**
Two threads calling `mysql.connector.connect()` at once, with the same
kwargs `DorisConnectionManager.open()` uses, each in a fresh process:
```python
import threading, mysql.connector
kw = dict(host=..., port=9030, user=..., password=...) # add use_pure=True
for the control
def go():
c = mysql.connector.connect(**kw); c.cursor().execute("SELECT 1");
c.close()
ts = [threading.Thread(target=go) for _ in range(2)]
[t.start() for t in ts]; [t.join() for t in ts]
```
One run of this exited 139 (`Segmentation fault (core dumped)`). To be
honest about the strength of this: it's rare in isolation. It crashed once in
about 240 C-extension runs, and 0 in 260 `use_pure=True` runs. So the
standalone repro shows the crash is real but isn't a reliable reproducer.
Inside dbt, with the adapter's threads, it's far more frequent.
**Why it matters for this issue**
`DorisConnectionManager.open()` doesn't set `use_pure`, and the profile
can't pass it, so the only workaround today is `threads: 1`. Forcing
`use_pure=True` as suggested above (or making it a profile option) would cover
this crash as well as the large-statement corruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]