[
https://issues.apache.org/jira/browse/PHOENIX-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356262#comment-15356262
]
Thomas D'Silva commented on PHOENIX-2995:
-----------------------------------------
[~jamestaylor] ,
[~mujtabachohan] and I discussed the latest results (with stats cached outside
of PTable). We think the time taken to write 1K rows increases while it is
creating the views because it ends up calling PMetaDataImpl.addTable which
clones the PMetaDataCache which takes a significant amount of time.
{code}
private static Map<PTableKey,PTableRef>
cloneMap(Map<PTableKey,PTableRef> tables, int expectedCapacity) {
Map<PTableKey,PTableRef> newTables =
newMap(Math.max(tables.size(),expectedCapacity));
// Copy value so that access time isn't changing anymore
for (PTableRef tableAccess : tables.values()) {
newTables.put(tableAccess.getTable().getKey(), new
PTableRef(tableAccess));
}
return newTables;
}
{code}
Do you know why we have to create a copy of the PTableRef ? It looks like after
calling PMetaDataImpl.addTable we always assign the new cache back to the
original reference.
> Write performance severely degrades with large number of views
> ---------------------------------------------------------------
>
> Key: PHOENIX-2995
> URL: https://issues.apache.org/jira/browse/PHOENIX-2995
> Project: Phoenix
> Issue Type: Bug
> Reporter: Mujtaba Chohan
> Assignee: Thomas D'Silva
> Labels: Argus
> Fix For: 4.9.0
>
> Attachments: create_view_and_upsert.png, upsert_rate.png
>
>
> Write performance for each 1K batch degrades significantly when there are
> *10K* views being written in random with default
> {{phoenix.client.maxMetaDataCacheSize}}. With all views created, upsert rate
> remains around 25 seconds per 1K batch i.e. ~2K rows/min upsert rate.
> When {{phoenix.client.maxMetaDataCacheSize}} is increased to 100MB+ then view
> does not need to get re-resolved and upsert rate gets back to normal ~60K
> rows/min.
> With *100K* views and {{phoenix.client.maxMetaDataCacheSize}} set to 1GB, I
> wasn't able create all 100K views as upsert time for each 1K batch keeps on
> steadily increasing.
> Following graph shows 1K batch upsert rate over time with variation of number
> of views. Rows are upserted to random views {{CREATE VIEW IF NOT EXISTS ...
> APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY=900000}} is executed before
> upsert statement.
> !upsert_rate.png!
> Base table is also created with {{APPEND_ONLY_SCHEMA = true,
> UPDATE_CACHE_FREQUENCY = 900000, AUTO_PARTITION_SEQ}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
