I observer that there is some extra mutations in batch for every my UPSERTs
For example if app call executeUpdate() only 5 times then on commit there will 
be "DEBUG MutationState:1046 - Sent batch of 10"
Can’t figure out where this extra mutations comes from and why.

This is mean that “useful” batch size is phoenix.mutate.batchSize / 2.

> * What does your table DDL look like?

    "d"."tg" VARCHAR,
    "d"."drip" VARCHAR,
    "d"."se" UNSIGNED_TINYINT,
    "d"."rle" UNSIGNED_TINYINT,
    "d"."dme" TIMESTAMP,
    "d"."dpa" TIMESTAMP,
    "d"."p" VARCHAR,
    "d"."pt" UNSIGNED_TINYINT,
    "d"."x" VARCHAR,
    "d"."pn" VARCHAR,
    "d"."b" VARCHAR,
    "d"."hc" VARCHAR ARRAY,
    "d"."ns" VARCHAR(16),
    "d"."tv" VARCHAR(10),
    "d"."vcp" VARCHAR,
    "d"."et" UNSIGNED_TINYINT,
    "d"."xoa" BINARY(16),
    "d"."j" VARCHAR

CREATE INDEX "IDX_CIS_O" ON "TABLE_CODES" ("d"."x", "d"."dme") 
INCLUDE("d"."tg", "d"."rle", "d"."pt" ... ) SALT_BUCKETS=30;
CREATE INDEX "IDX_CIS_PRID" ON "TABLE_CODES" ("d"."drip", "d"."dme") 
INCLUDE("d"."tg", "d"."rle", "d"."pt" ...) SALT_BUCKETS=30;

For my case SALT_BUCKET=30 every batch with default settings will carry only 50 
“useful” rows and they will be splitted across 30 servers, so every server will 
get only 1-2 rows.

> * How large is one mutation you're writing (in bytes)?

Any idea how to calculate it?
<https://phoenix.apache.org/metrics.html> will give me total mutations count 
and total size in bytes of batch. But as I mentioned before there is “extra” 
mutation that will corrupt statistics

> * How much data ends up being sent to a RegionServer in one RPC?
Where I can get this metric?

> On 3 Sep 2019, at 17:19, Josh Elser <els...@apache.org> wrote:
> Hey Alexander,
> Was just poking at the code for this: it looks like this is really just 
> determining the number of mutations that get "processed together" (as opposed 
> to a hard limit).
> Since you have done some work, I'm curious if you could generate some data to 
> help back up your suggestion:
> * What does your table DDL look like?
> * How large is one mutation you're writing (in bytes)?
> * How much data ends up being sent to a RegionServer in one RPC?
> You're right in that we would want to make sure that we're sending an 
> adequate amount of data to a RegionServer in an RPC, but this is tricky to 
> balance for all cases (thus, setting a smaller value to avoid sending batches 
> that are too large is safer).
> On 9/3/19 8:03 AM, Alexander Batyrshin wrote:
>>  Hello all,
>> 1) There is bug in documentation - http://phoenix.apache.org/tuning.html
>> phoenix.mutate.batchSize is not 1000, but only 100 by default
>> https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java#L164
>> Changed for https://issues.apache.org/jira/browse/PHOENIX-541
>> 2) I want to discuss this default value. From PHOENIX-541 
>> <https://issues.apache.org/jira/browse/PHOENIX-541> I read about issue with 
>> MR and wide rows (2MB per row) and it looks like rare case. But in most 
>> common cases we can get much better write perfomance with batchSize = 1000 
>> especially if it used with SALT table

Reply via email to