[
https://issues.apache.org/jira/browse/PHOENIX-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702447#comment-16702447
]
Lars Hofhansl edited comment on PHOENIX-5044 at 11/28/18 10:27 PM:
-------------------------------------------------------------------
Thanks [~elserj].
First let me prefix this by saying that I do *not* feel strongly about this.
With PHOENIX-5026 we (Salesforce) have everything we need (we can turn server
side mutations off).
It just seemed best to remove stuff that won't work in most scenarios (both
scale or not).
Lemme try to answer your questions:
* I was testing on a single node cluster (but with all services running). 16m
rows. This is representative because these effects scale *and* I was able to
have Phoenix timeout on a delete even with this little data - and now imagine
an even larger delete. The problem is the how the delete work is chunked. The
single node setup also let attach a profiler easily and thus rule out other
effects.
* Client and server were also on the same machine (so I did not measure the
effect of the network). Not all cores were used so this was not CPU due to
client and server being on the same machine. The slower the network between
client and server and better a server side solution will fare on comparison.
* I'm leaving the index related UPSERT/SELECTS in place. Aside from LOCAL
indexes they cannot be run on the server, though. (unless you turn on even
cross region server UPSERT/SELECT which I would advise against strongly)
* What I found was that HBase simply does not handle bulk mutations unless we
chunk them. The added client work and roundtrip seem to be more than outweighed
by the better pacing and more natural HBase use we get from client side
triggered mutations.
* (I guess what I am saying that server side mutation already are not working
to move data into HBase, so we're not removing anything)
* A followup meta-point is that even massive mutation work better when handled
from the client.
* We had outages (or at least brown-out) due to scan being stuck (well slow)
issuing mutations, thereby preventing HFiles from being collected, holding up
handler threads, etc, etc.
* As for bulk... In many scenario UPSERT/SELECT and DELETED are already run on
the client. The real bulk answer - as you suggest - is M/R or Spark, etc.
Probably paired with bulk HFile loading.
* (I think the Splice machine guys added a separate endpoint for this, where
they somehow can pre-prepare the data and then HBase just - somehow - maps it
in.)
As I said, I don't feel strong. PHOENIX-5026 is all I need.
was (Author: lhofhansl):
Thanks [~elserj].
First let me prefix this by saying that I do *not* feel strongly about this.
With PHOENIX-5026 we have everything we need (we can turn server side mutations
off).
It just seemed best to remove stuff that won't work in most scenarios (both
scale or not).
Lemme try to answer your questions:
* I was testing on a single node cluster (but with all services running). 16m
rows. This is representative because these effects scale *and* I was able to
have Phoenix timeout on a delete even with this little data - and now imagine
an even larger delete. The problem is the how the delete work is chunked. The
single node setup also let attach a profiler easily and thus rule out other
effects.
* Client and server were also on the same machine (so I did not measure the
effect of the network). Not all cores were used so this was not CPU due to
client and server being on the same machine. The slower the network between
client and server and better a server side solution will fare on comparison.
* I'm leaving the index related UPSERT/SELECTS in place. Aside from LOCAL
indexes they cannot be run on the server, though. (unless you turn on even
cross region server UPSERT/SELECT which I would advise against strongly)
* What I found was that HBase simply does not handle bulk mutations unless we
chunk them. The added client work and roundtrip seem to be more than outweighed
by the better pacing and more natural HBase use we get from client side
triggered mutations.
* (I guess what I am saying that server side mutation already are not working
to move data into HBase, so we're not removing anything)
* A followup meta-point is that even massive mutation work better when handled
from the client.
* We had outages (or at least brown-out) due to scan being stuck (well slow)
issuing mutations, thereby preventing HFiles from being collected, holding up
handler threads, etc, etc.
* As for bulk... In many scenario UPSERT/SELECT and DELETED are already run on
the client. The real bulk answer - as you suggest - is M/R or Spark, etc.
Probably paired with bulk HFile loading.
* (I think the Splice machine guys added a separate endpoint for this, where
they somehow can pre-prepare the data and then HBase just - somehow - maps it
in.)
As I said, I don't feel strong. PHOENIX-5026 is all I need.
> Remove server side mutation code from Phoenix
> ---------------------------------------------
>
> Key: PHOENIX-5044
> URL: https://issues.apache.org/jira/browse/PHOENIX-5044
> Project: Phoenix
> Issue Type: Task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Priority: Major
> Attachments: 5044-looksee-v2.txt, 5044-looksee-v3.txt,
> 5044-looksee.txt
>
>
> This is for *discussion*. Perhaps controversial.
> It generally seems to be a bad - if well-intentioned - idea to trigger
> mutations directly from the server. The main causes are UPSERT SELECT for the
> same table and DELETE FROM.
> IMHO, it's generally better to allow the client to handle this. There might
> be larger network overhead, but we get better chunking, better pacing, and
> behavior more in line with how HBase was intended to work.
> In PHOENIX-5026 I introduced a flag to disable server triggered mutations in
> the two cases mentioned above. I now think it's better to just remove the
> server code and also perform these from the client.
> (Note that server side reads - aggregation, filters, etc - are still insanely
> valuable and not affected by this)
> Let's discuss.
> [~tdsilva], [[email protected]], [~jamestaylor], [~vincentpoon], [~gjacoby]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
