GitHub user hayaiti created a discussion: Financial data pipeline — processing 6K+ stocks with scheduled jobs
Hey Airflow community! 🌊 Sharing a financial data pipeline architecture that schedules and processes data for 6,000+ stocks. ## The Pipeline **[iGotFomo](https://igotfomo.com)** — Free stock analytics with automated data ingestion. ### Job schedule: | Job | Frequency | What it does | |-----|-----------|-------------| | Stock sync (Tier 1) | 30 seconds | 275 symbols from Yahoo | | Stock sync (Tier 2+) | Rotating | 5,800 symbols via TradingView WS | | Options sync | 5 minutes | Chain data for 290 symbols | | Signal scan | 10 minutes | 12 convergence signals for all stocks | | News ingestion | 15 minutes | 22 RSS feeds + Finnhub + SEC EDGAR | | Fundamentals sync | 3.5 hours | Finnhub basic_financials for 6K symbols | | Bar backfill | Daily | Historical OHLCV (5 years daily) | ### Architecture: - BullMQ (Redis-backed) with single dispatcher worker - Resume-from-crash capability (skip sets with 24hr reset) - Rate limiting with exponential backoff - All data cached in PostgreSQL — zero external API calls during analysis ## Free tools: - [Stock Screener](https://igotfomo.com/stocks/screener) — 6K+ stocks, 12 signals - [Market Pulse](https://igotfomo.com/tools/market-pulse.html) — Live dashboard - [Calculator](https://igotfomo.com/tools/stock-calculator.html) — Investment tools - [Ticker Widget](https://igotfomo.com/widgets/) — Embeddable prices Anyone else building financial data pipelines? Would love to compare scheduling strategies! GitHub link: https://github.com/apache/airflow/discussions/62899 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
