[
https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546650#comment-15546650
]
Varun Saxena commented on YARN-5585:
------------------------------------
bq. I purposefully used VARIABLE_SIZE because prefix can be empty bytes
That's correct. Sorry, had missed it.
bq. Given your point-5 is valid, id_prefix is need to be stored in column and
give it back to user while reading. Basically intention is user can provide
fromEntityPrefix as filter.
When fromEntityPrefix is given as query param, we will construct the row key
using it. We do not necessarily need a column. We can use Result#getRow() and
using EntityRowKey#parseRowKey in parseEntity to fetch the prefix. Like below.
{code}
EntityRowKey rowKey = EntityRowKey.parseRowKey(result.getRow());
entity.setIdPrefix(rowKey.getEntityIdPrefix());
{code}
bq. After fetching 2 rows, user knows prefix is 2 , and gives fromEntityPrefix
as 2 for retrieving next batch. Then reader need not to scan rows from
beginning rather directly start scanning row-key prefixed with 2. And stop row
need to be calculated for entityType level i.e till prefix 4.
Ok...Got it.
But do we need to copy over code from HBase i.e.
Scan#calculateTheClosestNextRowKeyForPrefix for it. What we can do is as under:
{code}
// get the bytes for stop row
entityRowKeyPrefix = new
EntityRowKeyPrefix(context.getClusterId(),
context.getUserId(), context.getFlowName(),
context.getFlowRunId(),
context.getAppId(), context.getEntityType());
// set stop row
byte[] stopRow = entityRowKeyPrefix.getRowKeyPrefix();
stopRow[stopRow.length - 1] = 0xFF;
scan.setStopRow(stopRow);
{code}
This is because getRowKeyPrefix will give a byte array ending with
Separator#QUALIFIERS i.e. "!", which is equivalent to 0x21 in hex. QUALIFIERS
will never be ending in a string equivalent of 0xFF so we can safely set last
byte to 0xFF and set it as stop row.
> [Atsv2] Add a new filter fromId in REST endpoints
> -------------------------------------------------
>
> Key: YARN-5585
> URL: https://issues.apache.org/jira/browse/YARN-5585
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
> Reporter: Rohith Sharma K S
> Assignee: Rohith Sharma K S
> Priority: Critical
> Attachments: 0001-YARN-5585.patch, YARN-5585-workaround.patch,
> YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the
> applications. Along with those, it would be good to add new filter i.e fromId
> so that entities can be retrieved after the fromId.
> Current Behavior : Default limit is set to 100. If there are 1000 entities
> then REST call gives first/last 100 entities. How to retrieve next set of 100
> entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is
> no way to achieve this.
> So proposal is to have fromId in the filter like
> *getApps?limit=5&&fromId=app-5* which gives list of apps from app-6 to
> app-10.
> Since ATS is targeting large number of entities storage, it is very common
> use case to get next set of entities using fromId rather than querying all
> the entites. This is very useful for pagination in web UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
