[ https://issues.apache.org/jira/browse/DRILL-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17597234#comment-17597234 ]
ASF GitHub Bot commented on DRILL-8287: --------------------------------------- jnturton commented on code in PR #2633: URL: https://github.com/apache/drill/pull/2633#discussion_r957433471 ########## exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/values/ScalarListener.java: ########## @@ -76,4 +79,33 @@ protected void setArrayNull() { protected UserException typeConversionError(String jsonType) { return loader.typeConversionError(schema(), jsonType); } + + /** + * Adds a field's most recent value to the pagination map. This is necessary for the HTTP plugin + * for index or keyset pagination where the API transmits values in the results that are used to + * generate the next page. + * + * This data is only stored if the pagination map is defined, and has keys. Review Comment: Can this be rewritten in terms of generic column listeners rather than pagination and the HTTP plugin? ########## exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/parser/SimpleMessageParser.java: ########## @@ -66,11 +68,13 @@ public class SimpleMessageParser implements MessageParser { private final String[] path; + private final Map<String, Object> paginationFields; - public SimpleMessageParser(String dataPath) { + public SimpleMessageParser(String dataPath, Map<String, Object> paginationFields) { Review Comment: Can we rename "pagination" here too? > Add Support for Keyset Based Pagination > --------------------------------------- > > Key: DRILL-8287 > URL: https://issues.apache.org/jira/browse/DRILL-8287 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - HTTP > Affects Versions: 1.20.2 > Reporter: Charles Givre > Assignee: Charles Givre > Priority: Major > Fix For: 2.0.0 > > > Some APIs such as HubSpot use values in the result set to indicate whether > there are additional pages. This PR adds support for this kind of > pagination. Note that current implementation only works for JSON based APIs. -- This message was sent by Atlassian Jira (v8.20.10#820010)