[
https://issues.apache.org/jira/browse/DRILL-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469990#comment-17469990
]
ASF GitHub Bot commented on DRILL-8092:
---------------------------------------
cgivre commented on a change in pull request #2414:
URL: https://github.com/apache/drill/pull/2414#discussion_r779650192
##########
File path: contrib/storage-http/Pagination.md
##########
@@ -0,0 +1,38 @@
+# Auto Pagination in Drill
+Remote APIs frequently implement some sort of pagination as a way of limiting
results. However, if you are performing bulk data analysis, it is necessary to
reassemble the
+data into one larger dataset. Drill's auto-pagination features allow this to
happen in the background, so that the user will get clean data back.
+
+To use a paginator, you simply have to configure the paginator in the
connection for the particular API.
+
+## Offset Pagination
+Offset Pagination uses commands similar to SQL which has a `LIMIT` and an
`OFFSET`. With an offset paginator, let's say you want 200 records and the
maximum page size is 50
+records, the offset paginator will break up your query into 4 requests as
shown below:
+
+* myapi.com?limit=50&offset=0
+* myapi.com?limit=50?offset=50
+* myapi.com?limit=50&offset=100
+* myapi.com?limit=50&offset=150
+
+### Configuring Offset Pagination
+To configure an offset paginator, simply add the following to the
configuration for your connection.
+
+```json
+"paginator": {
+ "limitField": "<limit>",
Review comment:
I changed all the `fields` to `fooParam` here and elsewhere. I also
changed `maxPageSize` to `pageSize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Add Auto Pagination to HTTP Storage Plugin
> ------------------------------------------
>
> Key: DRILL-8092
> URL: https://issues.apache.org/jira/browse/DRILL-8092
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
> Affects Versions: 1.19.0
> Reporter: Charles Givre
> Assignee: Charles Givre
> Priority: Major
> Fix For: 1.20.0
>
>
> See github
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
