serem53 commented on issue #67546:
URL: https://github.com/apache/doris/issues/67546#issuecomment-7047456879

   A second, separate symptom of the C extension in dbt-doris, which also goes 
away with `use_pure=True` / `threads: 1`: **a segfault at startup when 
`threads` > 1.**
   
   **Environment**
   
   - dbt-core 1.12.2, dbt-doris 1.0.0, mysql-connector-python 9.6.0 (`HAVE_CEXT 
= True`)
   - Python 3.13.14, Debian 12 (bookworm), x86_64, run from an Airflow worker 
container
   - Apache Doris 4.0.3-rc03
   
   **What happens**
   
   With `threads: 2` in the profile, `dbt build` sometimes dies about 20 s in, 
before any model runs. There's no Python traceback and no dbt error. stdout 
stops after `Concurrency: 2 threads`, stderr has only:
   
   ```
   resource_tracker.py: UserWarning: resource_tracker: There appear to be 2 
leaked semaphore objects to clean up at shutdown
   ```
   
   `logs/dbt.log` always ends at the same point, with two threads opening their 
first connection at the same moment:
   
   ```
   [ThreadPool]: Acquiring new doris connection 'list_'
   [ThreadPool]: Acquiring new doris connection 'list_'
   [ThreadPool]: Using doris connection "list_"
   [ThreadPool]: Using doris connection "list_"
   [ThreadPool]: Opening a new connection, currently in state init
   [ThreadPool]: Opening a new connection, currently in state init
   <log ends>
   ```
   
   This hit 5 of 14 scheduled builds over three days. On `threads: 1` we had 
about 200 builds with the same versions and the same server, and none crashed 
this way.
   
   **Repro outside dbt**
   
   Two threads calling `mysql.connector.connect()` at once, with the same 
kwargs `DorisConnectionManager.open()` uses, each in a fresh process:
   
   ```python
   import threading, mysql.connector
   kw = dict(host=..., port=9030, user=..., password=...)  # add use_pure=True 
for the control
   def go():
       c = mysql.connector.connect(**kw); c.cursor().execute("SELECT 1"); 
c.close()
   ts = [threading.Thread(target=go) for _ in range(2)]
   [t.start() for t in ts]; [t.join() for t in ts]
   ```
   
   One run of this exited 139 (`Segmentation fault (core dumped)`). To be 
honest about the strength of this: it's rare in isolation. It crashed once in 
about 240 C-extension runs, and 0 in 260 `use_pure=True` runs. So the 
standalone repro shows the crash is real but isn't a reliable reproducer. 
Inside dbt, with the adapter's threads, it's far more frequent.
   
   **Why it matters for this issue**
   
   `DorisConnectionManager.open()` doesn't set `use_pure`, and the profile 
can't pass it, so the only workaround today is `threads: 1`. Forcing 
`use_pure=True` as suggested above (or making it a profile option) would cover 
this crash as well as the large-statement corruption.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to