betodealmeida opened a new pull request #15843: URL: https://github.com/apache/superset/pull/15843
### SUMMARY <!--- Describe the change below, including rationale and design decisions --> This PR introduces a new `ChartDataResultType`, `ChartDataResultType.POST_PROCESSED`. It allows us to request the data for a given chart post-processed in the same way the visualization does in Javascript. Eg, for this pivot table:  If we request the data for the chart we get the pre-processed (**unpivoted**) data. Doing a `GET` on "http://localhost:9000/api/v1/chart/72/data/?format=json"; returns: ```json { "result": [ { "cache_key": "d432a37512f63a88562744f4da62021e", "cached_dttm": null, "cache_timeout": 86400, "annotation_data": {}, "error": null, "is_cached": false, "query": "SELECT gender AS gender,\n state AS state,\n sum(num) AS \"Births\"\nFROM birth_names\nWHERE ds >= TO_TIMESTAMP('1921-07-22 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.US')\n AND ds < TO_TIMESTAMP('2021-07-22 11:56:04.000000', 'YYYY-MM-DD HH24:MI:SS.US')\nGROUP BY gender,\n state\nLIMIT 100", "status": "success", "stacktrace": null, "rowcount": 22, "colnames": [ "gender", "state", "Births" ], "coltypes": [ 1, 1, 0 ], "data": [ { "gender": "boy", "state": "PA", "Births": 2390275 }, { "gender": "girl", "state": "PA", "Births": 1615383 }, { "gender": "boy", "state": "OH", "Births": 2376385 }, { "gender": "girl", "state": "MI", "Births": 1326229 }, { "gender": "girl", "state": "FL", "Births": 1312593 }, { "gender": "boy", "state": "CA", "Births": 5430796 }, { "gender": "boy", "state": "other", "Births": 22044909 }, { "gender": "girl", "state": "OH", "Births": 1622814 }, { "gender": "girl", "state": "TX", "Births": 2313186 }, { "gender": "boy", "state": "NY", "Births": 3543961 }, { "gender": "boy", "state": "IL", "Births": 2357411 }, { "gender": "girl", "state": "CA", "Births": 3567754 }, { "gender": "girl", "state": "NY", "Births": 2280733 }, { "gender": "girl", "state": "IL", "Births": 1614427 }, { "gender": "girl", "state": "other", "Births": 15058341 }, { "gender": "boy", "state": "TX", "Births": 3311985 }, { "gender": "boy", "state": "FL", "Births": 1968060 }, { "gender": "boy", "state": "MA", "Births": 1285126 }, { "gender": "boy", "state": "MI", "Births": 1938321 }, { "gender": "boy", "state": "NJ", "Births": 1486126 }, { "gender": "girl", "state": "NJ", "Births": 992702 }, { "gender": "girl", "state": "MA", "Births": 842146 } ], "applied_filters": [], "rejected_filters": [] } ] } ``` Note that the results above have 22 rows and 3 columns: "gender", "state", and "Birth" (the metric). With this PR we can do a `GET` request to "http://localhost:9000/api/v1/chart/72/data/?format=json&type=post_processed"; to get: ```json { "result": [ { "cache_key": "d432a37512f63a88562744f4da62021e", "cached_dttm": null, "cache_timeout": 86400, "annotation_data": {}, "error": null, "is_cached": false, "query": "SELECT gender AS gender,\n state AS state,\n sum(num) AS \"Births\"\nFROM birth_names\nWHERE ds >= TO_TIMESTAMP('1921-07-22 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.US')\n AND ds < TO_TIMESTAMP('2021-07-22 12:03:21.000000', 'YYYY-MM-DD HH24:MI:SS.US')\nGROUP BY gender,\n state\nLIMIT 100", "status": "success", "stacktrace": null, "rowcount": 3, "colnames": [ "Births CA", "Births FL", "Births IL", "Births MA", "Births MI", "Births NJ", "Births NY", "Births OH", "Births PA", "Births TX", "Births other", "Births All" ], "coltypes": [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "data": [ { "Births CA": 5430796, "Births FL": 1968060, "Births IL": 2357411, "Births MA": 1285126, "Births MI": 1938321, "Births NJ": 1486126, "Births NY": 3543961, "Births OH": 2376385, "Births PA": 2390275, "Births TX": 3311985, "Births other": 22044909, "Births All": 22044909, "gender": "boy" }, { "Births CA": 3567754, "Births FL": 1312593, "Births IL": 1614427, "Births MA": 842146, "Births MI": 1326229, "Births NJ": 992702, "Births NY": 2280733, "Births OH": 1622814, "Births PA": 1615383, "Births TX": 2313186, "Births other": 15058341, "Births All": 15058341, "gender": "girl" }, { "Births CA": 5430796, "Births FL": 1968060, "Births IL": 2357411, "Births MA": 1285126, "Births MI": 1938321, "Births NJ": 1486126, "Births NY": 3543961, "Births OH": 2376385, "Births PA": 2390275, "Births TX": 3311985, "Births other": 22044909, "Births All": 22044909, "gender": "All" } ], "applied_filters": [], "rejected_filters": [] } ] } ``` Note that we now get the correct results, with 3 rows and many columns from the pivot. This functionality was added to support a very small subset of visualization types: 1. Pivot tables (v1) 2. Pivot tables (v2) 3. T-test tables Because of the small number of visualization types, I opted to do this by reimplementing the chart logic in Python. The alternative would be to run the visualization under Selenium and scrape the data, which is inefficient and still requires keeping the scraping logic up-to-date with the plugins. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF <!--- Skip this if not applicable --> ### TESTING INSTRUCTIONS <!--- Required! What steps can be taken to manually verify the changes? --> Create and save a pivot chart (not v2). Check the data at "http://localhost:9000/api/v1/chart/{chartid}/data/?format=json&type=post_processed"; and "http://localhost:9000/api/v1/chart/{chartid}/data/?format=json&type=full";. ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [ ] Has associated issue: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
