mai-nguyen-coinhako commented on issue #22487: URL: https://github.com/apache/superset/issues/22487#issuecomment-1428395013
Hi, I have updated `PyAthena` to v2.22.0 and test in Superset, which is deployed in Kubernetes, but it is still not working. The table list still does not show up in the UI of SQLab. In my case, I use Athena to query data from AWS Timestream database through its query federation feature. From following the stack trace in Superset logging, an event `request-created.athena.ListTableMetadata` was created, and it successfully received a Json response from Athena with all the table metadata in the database as below: ``` 2023-02-13 17:09:30,995:DEBUG:botocore.hooks:Event request-created.athena.ListTableMetadata: calling handler <function add_retry_headers at 0x7fe1cc285ca0> 2023-02-13 17:09:30,996:DEBUG:botocore.endpoint:Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://athena.ap-southeast-2.amazonaws.com/, headers={'X-Amz-Target': b'AmazonAthena.ListTableMetadata', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.26.69 Python/3.8.12 Linux/5.4.219-126.411.amzn2.x86_64 Botocore/1.29.69 PyAthena/2.22.0', 'X-Amz-Date': b'20230213T170930Z' 023-02-13 17:09:30,996:DEBUG:botocore.httpsession:Certificate path: /usr/local/lib/python3.8/site-packages/certifi/cacert.pem 2023-02-13 17:09:30,997:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): athena.ap-southeast-2.amazonaws.com:443 2023-02-13 17:09:34,597:DEBUG:urllib3.connectionpool:https://athena.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 200 13012 2023-02-13 17:09:34,602:DEBUG:botocore.parsers:Response headers: {'Date': 'Mon, 13 Feb 2023 17:09:34 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '13012', 'Connection': 'keep-alive', 'x-amzn-RequestId': '27544755-53ff-43bd-9fd1-6d79cba83684'} 2023-02-13 17:09:34,603:DEBUG:botocore.parsers:Response body: b'{"TableMetadataList":[{"Columns":[{"Name":"id","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"coin","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"jump_eth","Type":"varchar"},{"Name":"event_datetime","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"jump_btc","Type":"varchar"},{"Name":"version","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"curve_gen_calendar","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"jump_eth","Type":"varchar"},{"Name":"event_datetime","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"jump_btc","Type":"varchar"},{"Name":"version","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"curve_gen_calend ar_test","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"dbt_strikes","Type":"varchar"},{"Name":"maturity_date","Type":"varchar"},{"Name":"all_strikes","Type":"varchar"},{"Name":"dbt_bids","Type":"varchar"},{"Name":"sabr_vols","Type":"varchar"},{"Name":"dbt_dn_strike","Type":"varchar"},{"Name":"dbt_asks","Type":"varchar"},{"Name":"dbt_10c_strike","Type":"varchar"},{"Name":"dbt_vols","Type":"varchar"},{"Name":"dbt_25p_strike","Type":"varchar"},{"Name":"dbt_10p_strike","Type":"varchar"},{"Name":"coin","Type":"varchar"},{"Name":"dbt_25c_strike","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"curve_gen_plotting_params","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"coin","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::double","Type":"double"}],"Name":"curve_parameters","PartitionKe ys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"btc_currency","Type":"varchar"},{"Name":"eth_equity","Type":"varchar"},{"Name":"ethw_available_funds","Type":"varchar"},{"Name":"usdc_margin_balance","Type":"varchar"},{"Name":"usdc_initial_margin","Type":"varchar"},{"Name":"receive_notifications","Type":"varchar"},{"Name":"type","Type":"varchar"},{"Name":"ethw_margin_balance","Type":"varchar"},{"Name":"eth_balance","Type":"varchar"},{"Name":"sol_available_withdrawal_funds","Type":"varchar"},{"Name":"security_keys_enabled","Type":"varchar"},{"Name":"sol_available_funds","Type":"varchar"},{"Name":"eth_available_withdrawal_funds","Type":"varchar"},{"Name":"ethw_equity","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"login_enabled","Type":"varchar"},{"Name":"btc_equity","Type":"varchar"},{"Name":"btc_available_funds","Type":"varchar"},{"Name":"is_password","Type":"varchar"},{"Name":"security_keys_assignments","Type":"varchar"},{"Name":"usdc_equity","Type":"varchar"},{"Name": "eth_available_funds","Type":"varchar"},{"Name":"ethw_balance","Type":"varchar"},{"Name":"referrals_count","Type":"varchar"},{"Name":"ethw_initial_margin","Type":"varchar"},{"Name":"ethw_available_withdrawal_funds","Type":"varchar"},{"Name":"sol_equity","Type":"varchar"},{"Name":"btc_margin_balance","Type":"varchar"},{"Name":"usdc_available_funds","Type":"varchar"},{"Name":"sol_currency","Type":"varchar"},{"Name":"btc_initial_margin","Type":"varchar"},{"Name":"eth_currency","Type":"varchar"},{"Name":"proof_id","Type":"varchar"},{"Name":"btc_balance","Type":"varchar"},{"Name":"usdc_balance","Type":"varchar"},{"Name":"usdc_maintenance_margin","Type":"varchar"},{"Name":"email","Type":"varchar"},{"Name":"ethw_currency","Type":"varchar"},{"Name":"sol_margin_balance","Type":"varchar"},{"Name":"ethw_maintenance_margin","Type":"varchar"},{"Name":"btc_maintenance_margin","Type":"varchar"},{"Name":"proof_id_signature","Type":"varchar"},{"Name":"sol_balance","Type":"varchar"},{"Name":"eth_main tenance_margin","Type":"varchar"},{"Name":"system_name","Type":"varchar"},{"Name":"sol_initial_margin","Type":"varchar"},{"Name":"usdc_available_withdrawal_funds","Type":"varchar"},{"Name":"sol_maintenance_margin","Type":"varchar"},{"Name":"eth_initial_margin","Type":"varchar"},{"Name":"usdc_currency","Type":"varchar"},{"Name":"btc_available_withdrawal_funds","Type":"varchar"},{"Name":"eth_margin_balance","Type":"varchar"},{"Name":"username","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"deribit_account_info","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"leverage","Type":"varchar"},{"Name":"initial_margin","Type":"varchar"},{"Name":"realized_profit_loss","Type":"varchar"},{"Name":"settlement_price","Type":"varchar"},{"Name":"maintenance_margin","Type":"varchar"},{"Name":"delta","Type":"varchar"},{"Name":"index_price","Type":"varchar"},{"Name":"floating_profit_ loss_usd","Type":"varchar"},{"Name":"theta","Type":"varchar"},{"Name":"uid","Type":"varchar"},{"Name":"total_profit_loss","Type":"varchar"},{"Name":"direction","Type":"varchar"},{"Name":"open_orders_margin","Type":"varchar"},{"Name":"kind","Type":"varchar"},{"Name":"mark_price","Type":"varchar"},{"Name":"average_price","Type":"varchar"},{"Name":"estimated_liquidation_price","Type":"varchar"},{"Name":"interest_value","Type":"varchar"},{"Name":"size_currency","Type":"varchar"},{"Name":"floating_profit_loss","Type":"varchar"},{"Name":"size","Type":"varchar"},{"Name":"instrument_name","Type":"varchar"},{"Name":"average_price_usd","Type":"varchar"},{"Name":"realized_funding","Type":"varchar"},{"Name":"gamma","Type":"varchar"},{"Name":"vega","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"deribit_account_position","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"stats_v olume_usd","Type":"varchar"},{"Name":"stats_low","Type":"varchar"},{"Name":"settlement_price","Type":"varchar"},{"Name":"index_price","Type":"varchar"},{"Name":"open_interest","Type":"varchar"},{"Name":"best_ask_amount","Type":"varchar"},{"Name":"greeks_delta","Type":"varchar"},{"Name":"stats_high","Type":"varchar"},{"Name":"underlying_index","Type":"varchar"},{"Name":"greeks_gamma","Type":"varchar"},{"Name":"current_funding","Type":"varchar"},{"Name":"best_bid_amount","Type":"varchar"},{"Name":"greeks_theta","Type":"varchar"},{"Name":"funding_8h","Type":"varchar"},{"Name":"estimated_delivery_price","Type":"varchar"},{"Name":"interest_rate","Type":"varchar"},{"Name":"mark_iv","Type":"varchar"},{"Name":"state","Type":"varchar"},{"Name":"bid_iv","Type":"varchar"},{"Name":"timestamp","Type":"varchar"},{"Name":"underlying_price","Type":"varchar"},{"Name":"mark_price","Type":"varchar"},{"Name":"greeks_rho","Type":"varchar"},{"Name":"interest_value","Type":"varchar"},{"Name":"best_bid_pri ce","Type":"varchar"},{"Name":"greeks_vega","Type":"varchar"},{"Name":"stats_price_change","Type":"varchar"},{"Name":"max_price","Type":"varchar"},{"Name":"min_price","Type":"varchar"},{"Name":"instrument_name","Type":"varchar"},{"Name":"best_ask_price","Type":"varchar"},{"Name":"stats_volume","Type":"varchar"},{"Name":"last_price","Type":"varchar"},{"Name":"ask_iv","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"deribit_option_price","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"}],"Name":"deribit_transaction_log","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"total_pl","Type":"varchar"},{"Name":"options_session_upl","Type":"varchar"},{"Name":"initial_margin","Type":"varchar"},{"Name":"options_session_rpl","Type":"varchar"},{"Name":"referrer_id","Type":"varchar"},{"Name":"mainten ance_margin","Type":"varchar"},{"Name":"futures_session_upl","Type":"varchar"},{"Name":"futures_session_rpl","Type":"varchar"},{"Name":"ETH DVOL","Type":"varchar"},{"Name":"portfolio_margining_enabled","Type":"varchar"},{"Name":"type","Type":"varchar"},{"Name":"equity","Type":"varchar"},{"Name":"ETH Price","Type":"varchar"},{"Name":"options_pl","Type":"varchar"},{"Name":"projected_maintenance_margin","Type":"varchar"},{"Name":"fee_balance","Type":"varchar"},{"Name":"futures_pl","Type":"varchar"},{"Name":"security_keys_enabled","Type":"varchar"},{"Name":"balance","Type":"varchar"},{"Name":"projected_initial_margin","Type":"varchar"},{"Name":"projected_delta_total","Type":"varchar"},{"Name":"margin_balance","Type":"varchar"},{"Name":"currency","Type":"varchar"},{"Name":"delta_total_map","Type":"varchar"},{"Name":"available_withdrawal_funds","Type":"varchar"},{"Name":"delta_total","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"login_enabled","Type":"varchar"},{"Name":"email" ,"Type":"varchar"},{"Name":"limits","Type":"varchar"},{"Name":"options_gamma","Type":"varchar"},{"Name":"options_vega","Type":"varchar"},{"Name":"session_upl","Type":"varchar"},{"Name":"creation_timestamp","Type":"varchar"},{"Name":"interuser_transfers_enabled","Type":"varchar"},{"Name":"system_name","Type":"varchar"},{"Name":"session_rpl","Type":"varchar"},{"Name":"options_theta","Type":"varchar"},{"Name":"options_delta","Type":"varchar"},{"Name":"options_value","Type":"varchar"},{"Name":"available_funds","Type":"varchar"},{"Name":"username","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::double","Type":"double"}],"Name":"historical","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"jump_eth","Type":"varchar"},{"Name":"event_datetime","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"jump_btc","Type":"varchar"},{"Name":"version","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Na me":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"last_occurrence_test","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"gen_25rr","Type":"varchar"},{"Name":"dbt_atm","Type":"varchar"},{"Name":"dbt_ask","Type":"varchar"},{"Name":"maturity_date","Type":"varchar"},{"Name":"gen_25bfly","Type":"varchar"},{"Name":"atm_strike","Type":"varchar"},{"Name":"dbt_bid","Type":"varchar"},{"Name":"dbt_25rr","Type":"varchar"},{"Name":"gen_10bfly","Type":"varchar"},{"Name":"dbt_10bfly","Type":"varchar"},{"Name":"dbt_10rr","Type":"varchar"},{"Name":"dbt_25bfly","Type":"varchar"},{"Name":"gen_10rr","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"gen_atm","Type":"varchar"},{"Name":"maturity_date_ts","Type":"varchar"},{"Name":"coin","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"voltable","PartitionKeys":[],"TableType":"EXTERNAL"},{" Columns":[{"Name":"dbt_ask","Type":"varchar"},{"Name":"all_strikes","Type":"varchar"},{"Name":"gen_25bfly","Type":"varchar"},{"Name":"atm_strike","Type":"varchar"},{"Name":"dbt_bids","Type":"varchar"},{"Name":"dbt_bid","Type":"varchar"},{"Name":"gen_10bfly","Type":"varchar"},{"Name":"dbt_10bfly","Type":"varchar"},{"Name":"dbt_10rr","Type":"varchar"},{"Name":"dbt_vols","Type":"varchar"},{"Name":"gen_10rr","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"gen_atm","Type":"varchar"},{"Name":"dbt_strikes","Type":"varchar"},{"Name":"gen_25rr","Type":"varchar"},{"Name":"dbt_atm","Type":"varchar"},{"Name":"sabr_vols","Type":"varchar"},{"Name":"dbt_dn_strike","Type":"varchar"},{"Name":"dbt_25rr","Type":"varchar"},{"Name":"dbt_asks","Type":"varchar"},{"Name":"dbt_10c_strike","Type":"varchar"},{"Name":"dbt_25p_strike","Type":"varchar"},{"Name":"dbt_25bfly","Type":"varchar"},{"Name":"dbt_10p_strike","Type":"varchar"},{"Name":"coin","Type":"varchar"},{"Name":"dbt_25c_strike","Type":"var char"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"voltable_merged","PartitionKeys":[],"TableType":"EXTERNAL"},{"Columns":[{"Name":"gen_25rr","Type":"varchar"},{"Name":"dbt_atm","Type":"varchar"},{"Name":"dbt_ask","Type":"varchar"},{"Name":"maturity_date","Type":"varchar"},{"Name":"gen_25bfly","Type":"varchar"},{"Name":"atm_strike","Type":"varchar"},{"Name":"dbt_bid","Type":"varchar"},{"Name":"dbt_25rr","Type":"varchar"},{"Name":"gen_10bfly","Type":"varchar"},{"Name":"dbt_10bfly","Type":"varchar"},{"Name":"dbt_10rr","Type":"varchar"},{"Name":"dbt_25bfly","Type":"varchar"},{"Name":"gen_10rr","Type":"varchar"},{"Name":"id","Type":"varchar"},{"Name":"gen_atm","Type":"varchar"},{"Name":"maturity_date_ts","Type":"varchar"},{"Name":"coin","Type":"varchar"},{"Name":"measure_name","Type":"varchar"},{"Name":"time","Type":"timestamp"},{"Name":"measure_value::varchar","Type":"varchar"}],"Name":"voltable _ray_test","PartitionKeys":[],"TableType":"EXTERNAL"}]}' 2023-02-13 17:09:34,609:DEBUG:botocore.hooks:Event needs-retry.athena.ListTableMetadata: calling handler <botocore.retryhandler.RetryHandler object at 0x7fe148ed3d00> 2023-02-13 17:09:34,610:DEBUG:botocore.retryhandler:No retry needed. ``` It can be seen that there is a new Table Type `EXTERNAL` for Athena federation query. The fix from this PR [#403](https://github.com/laughingman7743/PyAthena/pull/403) in PyAthena only supports **EXTERNAL_TABLE** and **MANAGED_TABLE**, which is created via Spark when Glue is enabled as hivemeta store, though `EXTERNAL` are fully queryable in Athena. On the bright side, I still can query data following the below format: ``` Select * from catalog_name.database_name.table_name limit 10 ``` **How to reproduce the bug** 1. Create an AWS Timestream database with around 10 tables 2. Create a Athena data source 3. Create a federated query lambda 4. Check that it works from Athena 5. Add database in Superset 6. Go to SQLab, choose Athena database and its schema 7. Try to create a dataset in Superset ### Expected results 1. In SQLab, a list of tables should be seen from the drop-down list 2. A list of tables from which I can pick which one to use to create a dataset ### Actual results When creating a dataset, it loads for some time and eventually finishes with "no data" and no errors in the logs (after retrying some times) In SQLab, it loads for some time and eventually finishes with "no data" even though the API called to Athena TableMetadata is successful in the logs. ### Screenshots <img width="691" alt="image" src="https://user-images.githubusercontent.com/99168209/218532826-75d1856d-6937-4f01-8136-c30f63c634fe.png";> ### Environment (please complete the following information): 1. Browser type and version: Gecko/20100101 Firefox/109.0 2. Superset version: 2.0.1 3. Python version: Python 3.8.12 4. Node.js version: - 5. Any feature flags active: - GENERIC_CHART_AXES - ALERT_REPORTS - THUMBNAILS - GLOBAL_ASYNC_QUERIES - DASHBOARD_CACHE ### Checklist Make sure to follow these steps before submitting your issue - thank you! - [ ] I have checked the superset logs for python stacktraces and included it here as text if there are any. - [ ] I have reproduced the issue with at least the latest released version of superset. - [ ] I have checked the issue tracker for the same issue and I haven't found one similar. ### Additional context -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/99168209/218532826-75d1856d-6937-4f01-8136-c30f63c634fe.png"](https://user-images.githubusercontent.com/99168209/218532826-75d1856d-6937-4f01-8136-c30f63c634fe.png"){kind=link}