Hi Josh, Thanks for starting this discussion. Overall the design looks good to me!
Actually I have an internal implementation for logging user queries to text file over PQS. Let me explain our use cases. We wanted to find out following facts using our ugly hacks: - Running queries against PQS - Find out "WRONG" queries On our systems and applications, I've realized that It's hard to control unexpected large scans, full scans or running queries without indexes against very large table on HBase/Phoenix. Sometimes users forgot to add the "WHERE" clause for queries :-) And also we could not find an effective way to stop or cancel the wrong queries immediately. Therefore, we should find out the users and heavy queries from the query logs and then we noticed users about the impacts of their queries -- Please try this at home, not production cluster :-) Definitely it would be nice to have query logging out of the box! And if the features are implemented like your proposal, we can replace an ugly hacks with built-in feature from our cluster. Thanks, - Youngwoo On Fri, Mar 2, 2018 at 10:05 AM, Josh Elser <els...@apache.org> wrote: > Any feedback from folks? Not sure if the silence should be interpreted > as ambivalence or plain old being busy :) > > On Mon, Feb 26, 2018 at 4:57 PM, Josh Elser <els...@apache.org> wrote: > > Hiya, > > > > I wanted to share this little design doc with you about some feature work > > we've been thinking about. The following is a Google doc in which anyone > > should be allowed to comment. Feel free to comment there, or here on the > > thread. > > > > https://s.apache.org/phoenix-query-log > > > > The high-level goal is to create a construct in which Phoenix clients > will > > automatically serialize information about the queries they run to a table > > for retrospective analysis. Ideally, this information would be stored in > a > > Phoenix table. We want this data to help answer questions like: > > > > * What queries are running against my system > > * What specific queries started between 535AM and 620AM two days ago > > * What queries are user "bob" running > > * Are my user's queries effectively using the indexes in the system > > > > Anti-goals for include: > > > > * Cluster impact (computation/memory) usage of a query > > * Query performance may be slowed to ensure all data is serialized > > * A third-party service dedicated to ensuring query info is serialized > (in > > the event of client failure) > > > > Take a look at the document and let us know what you think please. I'm > happy > > to try to explain this in greater detail. > > > > - Josh (on behalf of myself, Ankit, Rajeshbabu, and Sergey) >
