GitHub user hayaiti created a discussion: Airflow DAGs for Stock Market Data
Pipeline
Sharing our data pipeline architecture for a stock market platform processing
6,000+ symbols.
## DAG Structure
```python
# Conceptual Airflow DAGs matching our BullMQ jobs
# DAG 1: Price Sync (every 30s during market hours)
stock_price_sync = DAG('stock_price_sync', schedule='*/30 * * * * *')
# DAG 2: OHLCV Bar Sync (hourly)
ohlcv_sync = DAG('ohlcv_bar_sync', schedule='0 * * * *')
# DAG 3: Options Chain Sync (every 5min)
options_sync = DAG('options_chain_sync', schedule='*/5 * * * *')
# DAG 4: News Ingestion (every 15min)
news_ingest = DAG('news_ingestion', schedule='*/15 * * * *')
# DAG 5: Convergence Scanner (every 30min)
convergence_scan = DAG('convergence_scan', schedule='*/30 * * * *')
# DAG 6: Earnings Analysis (daily)
earnings_analysis = DAG('earnings_analysis', schedule='0 6 * * *')
```
## Data Sources (All Free)
| Source | Data | Limit |
|--------|------|-------|
| Yahoo Finance | Quotes, history, options | ~2000/hr |
| SEC EDGAR | Filings, insider trades | 10/sec |
| Finnhub | Earnings, fundamentals | 60/min |
| CBOE/Nasdaq | Options flow | Scraping |
| 22 RSS feeds | Market news | No limit |
## Pipeline Output
- 6,137 stock quotes updated every 30s
- 2M+ OHLCV rows in PostgreSQL
- 12 convergence signal types computed
- AI earnings analysis via Claude
## Live Platform
- **[Stock Screener](https://igotfomo.com/stocks/screener)** — AI convergence
signals
- **[Market Pulse](https://igotfomo.com/tools/market-pulse.html)** — Live
dashboard
- **[Ticker Widget](https://igotfomo.com/widgets/)** — Embeddable prices
GitHub link: https://github.com/apache/airflow/discussions/62911
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
