kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-542431579
@mik-laj Agree with most of your comments, will update it by end of this
week.
-
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-535914795
I am going to squash this PR.
I have created a branch on my fork
(https://github.com/kaxil/airflow
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-532172943
> > * Do we want to let users use old versions of MySQL (and Mariadb that
doesn't support JSON with sqlal
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-530789821
Few things to do:
- [ ] Add
https://github.com/astronomer/airflow/commit/baf12f626e6d56dfde735faae
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-530785443
Few things to do:
- [ ] Add
https://github.com/astronomer/airflow/commit/baf12f626e6d56dfde735faae
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-529024391
@ashb @coufon With the above commit:
https://github.com/apache/airflow/pull/5743/commits/951cee34e6ecb60b
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-528982114
> That last point is interesting - cos at the SQL level we still need to
write it as a string. I wonder i
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-528697431
Some more testing:
For another trial, I am completely removing the need to load JSON from a str
by
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-528654768
Have done some more benchmarking:
![image](https://user-images.githubusercontent.com/8811558/64390
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-52749
> How about instead of a background thread (I'm wary of using threads in
python) could we instead query t
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-523847270
> fileloc is not stored in DB for subdag, because subdag does not have a row
in DB. subdag's fileloc is d
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-523416060
> Bug:
>
> ```
> [2019-08-21 12:19:17,625] {app.py:1891} ERROR - Exception on /task [GET]
> T
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-523239054
As far as I can tell, this PR is now working & ready for Code Review.
**Notes**:
- **Rendere
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-521358954
> > Pending Issues:
> >
> > * We still have the issue of `SerializedBaseOperator` being displayed i
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-520101071
Pending Issues:
- ~Add Timezone support to `serialized_dag` table~
- We still have the issue of
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-519723159
Just rebased
This is an automated messag
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-519647789
> hmm. It still failing on rat. This is strange. I am taking a look.
Thanks
--
17 matches
Mail list logo