Re: Question - HttpHook
Thank you very much, Matt! On Wed, Jan 18, 2017 at 5:11 PM Matt Hickmanwrote: > Couple things: > - You shouldn’t need to set port 443 > - Make sure the url starts with https:// in the connection settings > - To check your auth is working properly try something like `print > api_hook.get_conn(None).auth` > - My connection in airflow has a trailing / and when I run your code > below with the endpoint beginning with a “/“ I also get some html in the > response. Try changing `api_hook.run('/api/2.0/jobs/list’)` to > `api_hook.run('api/2.0/jobs/list’)`. > > Cheers, > Matt > > On January 18, 2017 at 4:48:19 PM, Hanneli Tavante (hannel...@gmail.com) > wrote: > > Thank you very much for the reply! > I set these parameters using the web interface. I tried several combos for > an HTTP type: > - setting host, login, password only > - setting schema field with https + setting host, login, password only > - setting port 443 + setting host, login, password only > - setting port 443 + schema field https + setting host, login, password > only > > for a certain my_dag.py, I run a PythonOperator calling a method that does > the following: > ``` api_hook = > HttpHook(http_conn_id='databricks_dcs_convert_json_to_parquet_raw_workflow_job_id', > method='GET') > > response = api_hook.run('/api/2.0/jobs/list') > ``` > > I also tried a complete version of run method: > ``` > response = api_hook.run('/api/2.0/jobs/list', None, 'application/json', > None) > > ``` > The problem is the same - I end up on the Login page. > > (I know my credentials are fine since I can perform a plain curl) > > It is odd since I succeed for other API integrations. > > Thank you very much for the help! > > > On Wed, Jan 18, 2017 at 4:38 PM Matt Hickman > wrote: > > Hanneli, > > We hit the data bricks api from Airflow. The connection info for our data > bricks instance is pretty simple: > > conn type: http > host: https://.cloud.databricks.com/ > login: > password: > > Knowing what error you get would be helpful! Also if you provide the code > your using for your hook. > > Cheers, > Matt > > On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannel...@gmail.com) > wrote: > > Hi! Could anybody help me? > I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8; > Databricks API 2.0 and Python 2.7.6 ) > Is there any example with that? It looks like I can't authenticate. I > created an HTTP connection on 'Connections', inserted my credentials and > pointed to the port 443, but I did not succeed. > Thanks! > >
Re: Question - HttpHook
Couple things: - You shouldn’t need to set port 443 - Make sure the url starts with https:// in the connection settings - To check your auth is working properly try something like `print api_hook.get_conn(None).auth` - My connection in airflow has a trailing / and when I run your code below with the endpoint beginning with a “/“ I also get some html in the response. Try changing `api_hook.run('/api/2.0/jobs/list’)` to `api_hook.run('api/2.0/jobs/list’)`. Cheers, Matt On January 18, 2017 at 4:48:19 PM, Hanneli Tavante (hannel...@gmail.com) wrote: Thank you very much for the reply! I set these parameters using the web interface. I tried several combos for an HTTP type: - setting host, login, password only - setting schema field with https + setting host, login, password only - setting port 443 + setting host, login, password only - setting port 443 + schema field https + setting host, login, password only for a certain my_dag.py, I run a PythonOperator calling a method that does the following: ``` api_hook = HttpHook(http_conn_id='databricks_dcs_convert_json_to_parquet_raw_workflow_job_id', method='GET') response = api_hook.run('/api/2.0/jobs/list') ``` I also tried a complete version of run method: ``` response = api_hook.run('/api/2.0/jobs/list', None, 'application/json', None) ``` The problem is the same - I end up on the Login page. (I know my credentials are fine since I can perform a plain curl) It is odd since I succeed for other API integrations. Thank you very much for the help! On Wed, Jan 18, 2017 at 4:38 PM Matt Hickmanwrote: Hanneli, We hit the data bricks api from Airflow. The connection info for our data bricks instance is pretty simple: conn type: http host: https://.cloud.databricks.com/ login: password: Knowing what error you get would be helpful! Also if you provide the code your using for your hook. Cheers, Matt On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannel...@gmail.com) wrote: Hi! Could anybody help me? I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8; Databricks API 2.0 and Python 2.7.6 ) Is there any example with that? It looks like I can't authenticate. I created an HTTP connection on 'Connections', inserted my credentials and pointed to the port 443, but I did not succeed. Thanks!
Re: Question - HttpHook
Hanneli, We hit the data bricks api from Airflow. The connection info for our data bricks instance is pretty simple: conn type: http host: https://.cloud.databricks.com/ login: password: Knowing what error you get would be helpful! Also if you provide the code your using for your hook. Cheers, Matt On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannel...@gmail.com) wrote: Hi! Could anybody help me? I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8; Databricks API 2.0 and Python 2.7.6 ) Is there any example with that? It looks like I can't authenticate. I created an HTTP connection on 'Connections', inserted my credentials and pointed to the port 443, but I did not succeed. Thanks!