[
https://issues.apache.org/jira/browse/PHOENIX-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421494#comment-15421494
]
James Taylor commented on PHOENIX-2995:
---------------------------------------
bq. Samarth had suggested using a ReadWriteLock instead of synchronizing on the
latestMetaDataLock object and using the read lock in the connect method and
write lock everywhere else .
What would this buy us?
bq. However if all tables don't fit in the cache, it might not be present. I
found this out in one of my tests that creates views using multiple base tables
with a smaller cache size.
I'd rather update the test than have to have these updateCache calls have to be
made all over the place. Let's file this JIRA as a potential issue, though
(realistically, the cache will always be big enough to fit the handful of
tables that might be necessary). Ideally, we should make at most one
updateCache call where we can pass multiple table names and then the client
should treat this is a single block without evicting any of the referenced
tables.
> Write performance severely degrades with large number of views
> ---------------------------------------------------------------
>
> Key: PHOENIX-2995
> URL: https://issues.apache.org/jira/browse/PHOENIX-2995
> Project: Phoenix
> Issue Type: Bug
> Reporter: Mujtaba Chohan
> Assignee: Thomas D'Silva
> Labels: Argus
> Fix For: 4.8.1
>
> Attachments: PHOENIX-2995-v2.patch, PHOENIX-2995-v3.patch,
> PHOENIX-2995-v4.patch, PHOENIX-2995.patch, create_view_and_upsert.png,
> image.png, image2.png, image3.png, upsert_rate.png
>
>
> Write performance for each 1K batch degrades significantly when there are
> *10K* views being written in random with default
> {{phoenix.client.maxMetaDataCacheSize}}. With all views created, upsert rate
> remains around 25 seconds per 1K batch i.e. ~2K rows/min upsert rate.
> When {{phoenix.client.maxMetaDataCacheSize}} is increased to 100MB+ then view
> does not need to get re-resolved and upsert rate gets back to normal ~60K
> rows/min.
> With *100K* views and {{phoenix.client.maxMetaDataCacheSize}} set to 1GB, I
> wasn't able create all 100K views as upsert time for each 1K batch keeps on
> steadily increasing.
> Following graph shows 1K batch upsert rate over time with variation of number
> of views. Rows are upserted to random views {{CREATE VIEW IF NOT EXISTS ...
> APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY=900000}} is executed before
> upsert statement.
> !upsert_rate.png!
> Base table is also created with {{APPEND_ONLY_SCHEMA = true,
> UPDATE_CACHE_FREQUENCY = 900000, AUTO_PARTITION_SEQ}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
