[
https://issues.apache.org/jira/browse/PHOENIX-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959728#comment-14959728
]
James Taylor commented on PHOENIX-2292:
---------------------------------------
New numbers with batching in place:
| | 100M narrow table (min) | 1B narrow table (min) | 1B wide table (min)
| Non MR | 10 | 76 | 511
| HFile MR | 17 | 161 | 1,375
| Direct HBase APIs | 24 | 84 | 1,450
| Batched Direct HBase APIs | - | - | 720
The performance difference is mainly attributed to there being less concurrency
due to restricting the number of Mappers running with leads to needing a second
set of Mappers required to be run to complete the job. The good news is that
based on testing by [~mujtabachohan], the cluster is still usable wrt querying,
showing an impact in the 20-30% higher query times, and additional index builds
have no adverse impact as they just get queued behind the first job.
> Improve performance of direct HBase API index build
> ---------------------------------------------------
>
> Key: PHOENIX-2292
> URL: https://issues.apache.org/jira/browse/PHOENIX-2292
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: James Taylor
> Assignee: Ravi Kishore Valeti
> Attachments: PHOENIX-2292.patch
>
>
> The direct HBase API index build _should_ be almost as fast as the native
> Phoenix index build, but we're seeing a big difference:
> | | 100M narrow table (min) | 1B narrow table (min) | 1B wide table (min)
> | Non MR | 10 | 76 | 511
> | HFile MR | 17 | 161 | 1,375
> | Direct HBase APIs | 24 | 84 | 1,450
> These results are for a 8 node cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
