[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750559#comment-17750559
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre merged PR #2747:
URL: https://github.com/apache/drill/pull/2747
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750049#comment-17750049
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1661290803
> @LYCJeff Could you please rebase on current master. Other than that, LGTM
+1.
>
> Sorry this took so long. Thank you very much for this contribution!
@cgivre I have updated, thanks for your review.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749982#comment-17749982
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1660866785
@LYCJeff Could you please rebase on current master. Other than that, LGTM
+1. Sorry this took so long.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749520#comment-17749520
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1659744329
@cgivre Is there anything else I should do for this PR?
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713509#comment-17713509
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on code in PR #2747:
URL: https://github.com/apache/drill/pull/2747#discussion_r1169760773
##
contrib/storage-http/README.md:
##
@@ -15,6 +15,7 @@ To configure the plugin, create a new storage plugin, and add
the following conf
{
"type": "http",
"cacheResults": true,
+ "useLegacyRequestParamSyntax": false,
Review Comment:
Sure, I'll finish it soon.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713505#comment-17713505
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on code in PR #2747:
URL: https://github.com/apache/drill/pull/2747#discussion_r1169754553
##
contrib/storage-http/README.md:
##
@@ -138,11 +140,26 @@ key2=value2"
```
`postBodyLocation`: If the API uses the `POST` method, you can send
parameters in several different ways:
-* `query_string`: Parameters from the query are pushed down to the query
string. Static parameters are pushed to the post body.
Review Comment:
In the enhanced mode, default position is `json_body`.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713350#comment-17713350
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre commented on code in PR #2747:
URL: https://github.com/apache/drill/pull/2747#discussion_r1169428023
##
contrib/storage-http/README.md:
##
@@ -138,11 +140,26 @@ key2=value2"
```
`postBodyLocation`: If the API uses the `POST` method, you can send
parameters in several different ways:
-* `query_string`: Parameters from the query are pushed down to the query
string. Static parameters are pushed to the post body.
Review Comment:
If the user enables this feature, are they required to specify the
destination of the supplied parameters? What happens if they don't?
##
contrib/storage-http/README.md:
##
@@ -15,6 +15,7 @@ To configure the plugin, create a new storage plugin, and add
the following conf
{
"type": "http",
"cacheResults": true,
+ "useLegacyRequestParamSyntax": false,
Review Comment:
See comment. I'd like to make the default behavior what Drill currently
does and have the user explicitly enable this functionality. Could we rename
this param `advanced` or `enhanced` and thus change the boolean to
`enableEnhancedParamSyntax`?
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713349#comment-17713349
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1512348212
@LYCJeff
Thank you for submitting this and I'm sorry the review is taking so long.
This is a potentially very breaking change and that's why I've had to give this
so much thought. I've also been quite busy. Ok...
Let's get this merged.
Here are my final requests:
1. Please rebase on current master
2. Can we make the default behavior what Drill currently does and change
the boolean parameter to something like `enablePostHeadersInWhere` or something
that uses the word `enable`?
3. Make the default behavior to use the current implementation.
Once that's done, I'm good with merging.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700914#comment-17700914
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1471106650
> @LYCJeff Thanks for making these changes. I have a few questions:
>
> 1. Are you certain that these filters are in fact being pushed down as
intended?
> 2. I'm really concerned about what would happen if a user aliased a data
source as `header` or `tail`.
>
> IE:
>
> ```sql
> SELECT ...
> FROM api.foo
> INNER JOIN dfs.`tail.csv` AS tail
> ON tail.id = foo.id
> WHERE tail.name = 'something'
> ```
>
> Do we know how this would be interpreted?
Well, we actually need to recognize `header.xxx` as a whole parameter name,
so we need to use back quotes. Only then can it be pushed normally, so these
prefixes are not confused with data source aliases.
If the `name` in your example above is an argument to the `foo` api, it
should be written as follows.
```sql
SELECT ...
FROM api.foo
INNER JOIN dfs.`tail.csv` AS tail
ON tail.id=foo.id
WHERE `tail.name` = 'something'
```
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700907#comment-17700907
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1471002379
@LYCJeff Thanks for making these changes. I have a few questions:
1. Are you certain that these filters are in fact being pushed down as
intended?
2. I'm really concerned about what would happen if a user aliased a data
source as `header` or `tail`.
IE:
```sql
SELECT ...
FROM api.foo
INNER JOIN dfs.`tail.csv` AS tail
ON tail.id = foo.id
WHERE tail.name = 'something'
```
Do we know how this would be interpreted?
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695578#comment-17695578
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1451555417
> Two ideas
>
> 1. Since we won't backport this PR and it will only go out in the next
major release, some breakage inside a plugin is probably something that can be
swallowed.
> 2. If it is still desired to preserve the ability to use the existing
syntax in Drill 1.22 and beyond then a storage config option like
`"useLegacyRequestParmSyntax": true` could be added for users who want it.
@jnturton @cgivre That's a good idea without confusing old and new syntax,
although it requires existing users to make small additions to the
configuration. If it is acceptable to you, I will take some time to add a
configuration item in the near future.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695531#comment-17695531
]
ASF GitHub Bot commented on DRILL-8393:
---
jnturton commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1451459573
Two ideas
1. Since we won't backport this PR and it will only go out in the next major
release, some breakage inside a plugin is probably something that can be
swallowed.
2. If it is still desired to preserve the ability to use the existing syntax
in Drill 1.22 and beyond then a storage config option like
`"useLegacyRequestParmSyntax": true` could be added for users who want it.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694879#comment-17694879
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1449446242
> @LYCJeff I really like the functionality here, but I am concerned that
this is a breaking change and will affect existing Drill users. Also, it adds
effectively new syntax to the SQL queries.
@cgivre At this point, I can pass the unprefixed parameters in their place
by default, the way they were. This minimizes the impact on existing users,
except in the following cases. For example, the argument that the user passed
into the request body was called `header.xxx`, but now needs to be rewritten as
`body.header.xxx`, otherwise the argument will be passed into the request
header.
In addition, a problem that had been fixed would reappear. The argument that
is passed to the url path is also passed to the end of the url, which has been
clearly distinguished since I changed it.
Let me know if you think this is more friendly to existing users, then I'll
move in this direction.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694615#comment-17694615
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1448388913
@LYCJeff
I really like the functionality here, but I am concerned that this is a
breaking change and will affect existing Drill users. Also, it adds
effectively new syntax to the SQL queries.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688003#comment-17688003
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1428198156
@cgivre I'm not sure if this change is appropriate. I'd love to hear your
opinion.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
> Fix For: 1.21.0
>
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683705#comment-17683705
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff opened a new pull request, #2747:
URL: https://github.com/apache/drill/pull/2747
# [DRILL-8393](https://issues.apache.org/jira/browse/DRILL-8393): Allow
parameters to be passed to headers through SQL in WHERE clause
## Description
Allow parameters to be passed to headers through SQL in WHERE clause. Use
the _params_ configuration item to control what parameters are allowed in,
passing them into the body and header depending on the prefix.
Config:
`{ "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
"params": ["body.lat", "body.lng", "body.date", "header.header1"],
"parameterLocation": "json_body" }`
SQL Query:
```
SELECT * FROM api.sunrise
WHERE `body.lat` = 36.7201600
AND `body.lng` = -4.4203400
AND `body.date` = '2019-10-02'
AND `header.header1` = 'value1';
```
Post body:
`{ "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}`
Headers:
`{ "header1": "value1", ……}`
## Documentation
The _params_ configuration mode and SQL parameter passing mode need to be
described in the document.
## Testing
The _params_ configuration and SQL in the original unit test have been
changed and have passed this part of the unit test.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
> Fix For: 1.21.0
>
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
