Also, they're relying on Phoenix to do secondary index updates for them.
Obviously, you can do this faster than Phoenix can if you know the exact
use-case.
On 7/12/18 6:31 PM, Pedro Boado wrote:
A tip for performance is reusing the same preparedStatement , just
clearParameters() , set values and executeUpdate() over and over again.
Don't close the statement or connections after each upsert. Also, I
haven't seen any noticeable benefit on using jdbc batches as Phoenix
controls batching by when commit() is called.
Keep an eye on not calling commit after every executeUpdate (that's a
real performance killer) . Batch commits in every ~1k upserts .
Also that attempt of asynchronous code is probably another performance
killer. Are you creating a new Runnable per database write and opening
and closing dB connections per write? Just spawn a few threads (5 to 10,
if client cpu is not maxed keep increasing it) and send upserts in a for
loop reusing preparedStatement and connections.
With a cluster that size I would expect seeing tens of thousands of
writes per second.
Finally have you checked that all RS receive same traffic ?
On Thu, 12 Jul 2018, 23:10 Pedro Boado, <pedro.bo...@gmail.com
<mailto:pedro.bo...@gmail.com>> wrote:
I believe it's related to your client code - In our use case we do
easily 15k writes/sec in a cluster lower specced than yours.
Check that your jdbc connection has autocommit off so Phoenix can
batch writes and that table has a reasonable UPDATE_CACHE_FREQUENCY
( more than 60000 ).
On Thu, 12 Jul 2018, 21:54 alchemist, <alchemistsrivast...@gmail.com
<mailto:alchemistsrivast...@gmail.com>> wrote:
Thanks a lot for your help.
Our test is inserting new rows individually. For our use case,
we are
benchmarking that we could be able to get 10,000 new rows in a
minute, using
a cluster of writers if needed.
When executing the inserts with Phoenix API (UPSERT) we have
been able to
get up to 6,000 new rows per minute.
We changed our test to perform the inserts individually using
the HBase API
(Put) rather than Phoenix API (UPSERT) and got an improvement of
more than
10x. (up to 60,000 rows per minute).
What would explain this difference? I assume that in both cases
HBase must
grab the locks individually in the same way.
--
Sent from: http://apache-phoenix-user-list.1124778.n5.nabble.com/
