RE: Phoenix monitoring
JMX interfaces are fine, too. Thank you, Gunnar Download a free version of HPDSM, a unified big-data administration tool for Vertica and Hadoop at: HP DSM Downloadhttps://vertica.hpwsportal.com/#/Category/%7B%22categoryId%22%3A10185%7D/Show “People don’t know what they want until you show it to them… Our task is to read things that are not yet on the page.” — Steve Jobs From: Jain, Rohit (Trafodion) Sent: Wednesday, February 11, 2015 11:54 AM To: Nick Dimiduk; dev@phoenix.apache.org Cc: Tapper, Gunnar Subject: RE: Phoenix monitoring Thanks Nick! Yes Data Services Manager (DSM) does provide the kind of perspective Pari is looking for but not for Phoenix. Actually, if Phoenix is providing metrics and information in a repository or via a REST interface, on queries, elapsed times, etc. then certainly it could be incorporated into DSM. Trafodion is doing that now in R1.0 and we are working on having DSM provide full support for Trafodion along with complete support for HBase (some of it being already there), in order to provide a comprehensive view of all workloads from HDFS, HBase, to SQL. We did have plans to provide a SDK so that data services like Phoenix could be easily plugged into DSM. Not a hard task – just time and effort ☺. If Pari is interested, we can certainly demo the tool and go from there based on interest. Rohit From: Nick Dimiduk [mailto:ndimi...@gmail.com] Sent: Wednesday, February 11, 2015 9:01 AM To: dev@phoenix.apache.orgmailto:dev@phoenix.apache.org Cc: Jain, Rohit (Trafodion) Subject: Re: Phoenix monitoring Hi Pari, I'm not aware of a Phoenix-aware, end-to-end solution here. You can probably write a custom collector for a Phenox application that can report into OpenTSDB; I think JDBC metrics are available via JMX. That would be a great addition for that project! I've also seen a product demo from HP that does a lot of this for other tools on HBase, though I don't think it supports Phoenix yet (cc Rohit). Thanks, Nick On Wednesday, February 11, 2015, Pariksheet Barapatre pbarapa...@gmail.commailto:pbarapa...@gmail.com wrote: Hello All, Can you suggest good monitoring tool for phoenix. Our QA hbase cluster is crashing randomly. We wanted to know stats like which query causing the issue and how many queries running on that particular time window, resource consumption of query etc. Cheers Pari
Re: Phoenix monitoring
Hi Rohit, I am interested in knowing more about DSM and its capabilities. I have some questions: 1) Phoenix logs the traces in a Phoenix table. Can a Phoenix table work as a DSM repository? 2) Does DSM expect data to be in a particular format or does it provide a means to plug in you own data puller (JDBC, REST etc)? 3) Does it provide an analytical engine that can roll up data (among other operations) and slice and dice it on various dimensions? 4) Is it open sourced? From the email it sounded like it isn't. Thanks, Samarth On Wed, Feb 11, 2015 at 11:11 AM, Tapper, Gunnar gunnar.tap...@hp.com wrote: JMX interfaces are fine, too. Thank you, Gunnar Download a free version of HPDSM, a unified big-data administration tool for Vertica and Hadoop at: HP DSM Download https://vertica.hpwsportal.com/#/Category/%7B%22categoryId%22%3A10185%7D/Show “People don’t know what they want until you show it to them… Our task is to read things that are not yet on the page.” — Steve Jobs From: Jain, Rohit (Trafodion) Sent: Wednesday, February 11, 2015 11:54 AM To: Nick Dimiduk; dev@phoenix.apache.org Cc: Tapper, Gunnar Subject: RE: Phoenix monitoring Thanks Nick! Yes Data Services Manager (DSM) does provide the kind of perspective Pari is looking for but not for Phoenix. Actually, if Phoenix is providing metrics and information in a repository or via a REST interface, on queries, elapsed times, etc. then certainly it could be incorporated into DSM. Trafodion is doing that now in R1.0 and we are working on having DSM provide full support for Trafodion along with complete support for HBase (some of it being already there), in order to provide a comprehensive view of all workloads from HDFS, HBase, to SQL. We did have plans to provide a SDK so that data services like Phoenix could be easily plugged into DSM. Not a hard task – just time and effort ☺. If Pari is interested, we can certainly demo the tool and go from there based on interest. Rohit From: Nick Dimiduk [mailto:ndimi...@gmail.com] Sent: Wednesday, February 11, 2015 9:01 AM To: dev@phoenix.apache.orgmailto:dev@phoenix.apache.org Cc: Jain, Rohit (Trafodion) Subject: Re: Phoenix monitoring Hi Pari, I'm not aware of a Phoenix-aware, end-to-end solution here. You can probably write a custom collector for a Phenox application that can report into OpenTSDB; I think JDBC metrics are available via JMX. That would be a great addition for that project! I've also seen a product demo from HP that does a lot of this for other tools on HBase, though I don't think it supports Phoenix yet (cc Rohit). Thanks, Nick On Wednesday, February 11, 2015, Pariksheet Barapatre pbarapa...@gmail.commailto:pbarapa...@gmail.com wrote: Hello All, Can you suggest good monitoring tool for phoenix. Our QA hbase cluster is crashing randomly. We wanted to know stats like which query causing the issue and how many queries running on that particular time window, resource consumption of query etc. Cheers Pari
Re: Phoenix monitoring
Thanks All for your comments. So shall we wait for 4.3 to be available. We are using 4.2 right now and when we enabled tracing it already crashed down the HBase. Also, what is exact release date of 4.3. Few more doubts from my side - 1) What is best way to upgrade Phoenix. At HBase level, we can easily replace old jar with new one and restart the HBase. But there are many Java project which uses phoenix jars and they also need to be updated/compiled. Any best practices. 2) Is there any development/JIRA on creating workload management system where we can set/upgrade/downgrade the priority query level/user level, set decision parameters to abort phoenix query. Many Thanks Pari On 12 February 2015 at 08:19, Samarth Jain samarth.j...@gmail.com wrote: Hi Rohit, I am interested in knowing more about DSM and its capabilities. I have some questions: 1) Phoenix logs the traces in a Phoenix table. Can a Phoenix table work as a DSM repository? 2) Does DSM expect data to be in a particular format or does it provide a means to plug in you own data puller (JDBC, REST etc)? 3) Does it provide an analytical engine that can roll up data (among other operations) and slice and dice it on various dimensions? 4) Is it open sourced? From the email it sounded like it isn't. Thanks, Samarth On Wed, Feb 11, 2015 at 11:11 AM, Tapper, Gunnar gunnar.tap...@hp.com wrote: JMX interfaces are fine, too. Thank you, Gunnar Download a free version of HPDSM, a unified big-data administration tool for Vertica and Hadoop at: HP DSM Download https://vertica.hpwsportal.com/#/Category/%7B%22categoryId%22%3A10185%7D/Show “People don’t know what they want until you show it to them… Our task is to read things that are not yet on the page.” — Steve Jobs From: Jain, Rohit (Trafodion) Sent: Wednesday, February 11, 2015 11:54 AM To: Nick Dimiduk; dev@phoenix.apache.org Cc: Tapper, Gunnar Subject: RE: Phoenix monitoring Thanks Nick! Yes Data Services Manager (DSM) does provide the kind of perspective Pari is looking for but not for Phoenix. Actually, if Phoenix is providing metrics and information in a repository or via a REST interface, on queries, elapsed times, etc. then certainly it could be incorporated into DSM. Trafodion is doing that now in R1.0 and we are working on having DSM provide full support for Trafodion along with complete support for HBase (some of it being already there), in order to provide a comprehensive view of all workloads from HDFS, HBase, to SQL. We did have plans to provide a SDK so that data services like Phoenix could be easily plugged into DSM. Not a hard task – just time and effort ☺. If Pari is interested, we can certainly demo the tool and go from there based on interest. Rohit From: Nick Dimiduk [mailto:ndimi...@gmail.com] Sent: Wednesday, February 11, 2015 9:01 AM To: dev@phoenix.apache.orgmailto:dev@phoenix.apache.org Cc: Jain, Rohit (Trafodion) Subject: Re: Phoenix monitoring Hi Pari, I'm not aware of a Phoenix-aware, end-to-end solution here. You can probably write a custom collector for a Phenox application that can report into OpenTSDB; I think JDBC metrics are available via JMX. That would be a great addition for that project! I've also seen a product demo from HP that does a lot of this for other tools on HBase, though I don't think it supports Phoenix yet (cc Rohit). Thanks, Nick On Wednesday, February 11, 2015, Pariksheet Barapatre pbarapa...@gmail.commailto:pbarapa...@gmail.com wrote: Hello All, Can you suggest good monitoring tool for phoenix. Our QA hbase cluster is crashing randomly. We wanted to know stats like which query causing the issue and how many queries running on that particular time window, resource consumption of query etc. Cheers Pari -- Cheers, Pari
Re: Phoenix monitoring
w.r.t. release of 4.3, see this thread: http://search-hadoop.com/m/lz2la1mCuXS Cheers On Wed, Feb 11, 2015 at 9:46 PM, Pariksheet Barapatre pbarapa...@gmail.com wrote: Thanks All for your comments. So shall we wait for 4.3 to be available. We are using 4.2 right now and when we enabled tracing it already crashed down the HBase. Also, what is exact release date of 4.3. Few more doubts from my side - 1) What is best way to upgrade Phoenix. At HBase level, we can easily replace old jar with new one and restart the HBase. But there are many Java project which uses phoenix jars and they also need to be updated/compiled. Any best practices. 2) Is there any development/JIRA on creating workload management system where we can set/upgrade/downgrade the priority query level/user level, set decision parameters to abort phoenix query. Many Thanks Pari On 12 February 2015 at 08:19, Samarth Jain samarth.j...@gmail.com wrote: Hi Rohit, I am interested in knowing more about DSM and its capabilities. I have some questions: 1) Phoenix logs the traces in a Phoenix table. Can a Phoenix table work as a DSM repository? 2) Does DSM expect data to be in a particular format or does it provide a means to plug in you own data puller (JDBC, REST etc)? 3) Does it provide an analytical engine that can roll up data (among other operations) and slice and dice it on various dimensions? 4) Is it open sourced? From the email it sounded like it isn't. Thanks, Samarth On Wed, Feb 11, 2015 at 11:11 AM, Tapper, Gunnar gunnar.tap...@hp.com wrote: JMX interfaces are fine, too. Thank you, Gunnar Download a free version of HPDSM, a unified big-data administration tool for Vertica and Hadoop at: HP DSM Download https://vertica.hpwsportal.com/#/Category/%7B%22categoryId%22%3A10185%7D/Show “People don’t know what they want until you show it to them… Our task is to read things that are not yet on the page.” — Steve Jobs From: Jain, Rohit (Trafodion) Sent: Wednesday, February 11, 2015 11:54 AM To: Nick Dimiduk; dev@phoenix.apache.org Cc: Tapper, Gunnar Subject: RE: Phoenix monitoring Thanks Nick! Yes Data Services Manager (DSM) does provide the kind of perspective Pari is looking for but not for Phoenix. Actually, if Phoenix is providing metrics and information in a repository or via a REST interface, on queries, elapsed times, etc. then certainly it could be incorporated into DSM. Trafodion is doing that now in R1.0 and we are working on having DSM provide full support for Trafodion along with complete support for HBase (some of it being already there), in order to provide a comprehensive view of all workloads from HDFS, HBase, to SQL. We did have plans to provide a SDK so that data services like Phoenix could be easily plugged into DSM. Not a hard task – just time and effort ☺. If Pari is interested, we can certainly demo the tool and go from there based on interest. Rohit From: Nick Dimiduk [mailto:ndimi...@gmail.com] Sent: Wednesday, February 11, 2015 9:01 AM To: dev@phoenix.apache.orgmailto:dev@phoenix.apache.org Cc: Jain, Rohit (Trafodion) Subject: Re: Phoenix monitoring Hi Pari, I'm not aware of a Phoenix-aware, end-to-end solution here. You can probably write a custom collector for a Phenox application that can report into OpenTSDB; I think JDBC metrics are available via JMX. That would be a great addition for that project! I've also seen a product demo from HP that does a lot of this for other tools on HBase, though I don't think it supports Phoenix yet (cc Rohit). Thanks, Nick On Wednesday, February 11, 2015, Pariksheet Barapatre pbarapa...@gmail.commailto:pbarapa...@gmail.com wrote: Hello All, Can you suggest good monitoring tool for phoenix. Our QA hbase cluster is crashing randomly. We wanted to know stats like which query causing the issue and how many queries running on that particular time window, resource consumption of query etc. Cheers Pari -- Cheers, Pari
Re: Phoenix monitoring
Thanks Ted. Cheers Pari On 12 February 2015 at 11:38, Ted Yu yuzhih...@gmail.com wrote: w.r.t. release of 4.3, see this thread: http://search-hadoop.com/m/lz2la1mCuXS Cheers On Wed, Feb 11, 2015 at 9:46 PM, Pariksheet Barapatre pbarapa...@gmail.com wrote: Thanks All for your comments. So shall we wait for 4.3 to be available. We are using 4.2 right now and when we enabled tracing it already crashed down the HBase. Also, what is exact release date of 4.3. Few more doubts from my side - 1) What is best way to upgrade Phoenix. At HBase level, we can easily replace old jar with new one and restart the HBase. But there are many Java project which uses phoenix jars and they also need to be updated/compiled. Any best practices. 2) Is there any development/JIRA on creating workload management system where we can set/upgrade/downgrade the priority query level/user level, set decision parameters to abort phoenix query. Many Thanks Pari On 12 February 2015 at 08:19, Samarth Jain samarth.j...@gmail.com wrote: Hi Rohit, I am interested in knowing more about DSM and its capabilities. I have some questions: 1) Phoenix logs the traces in a Phoenix table. Can a Phoenix table work as a DSM repository? 2) Does DSM expect data to be in a particular format or does it provide a means to plug in you own data puller (JDBC, REST etc)? 3) Does it provide an analytical engine that can roll up data (among other operations) and slice and dice it on various dimensions? 4) Is it open sourced? From the email it sounded like it isn't. Thanks, Samarth On Wed, Feb 11, 2015 at 11:11 AM, Tapper, Gunnar gunnar.tap...@hp.com wrote: JMX interfaces are fine, too. Thank you, Gunnar Download a free version of HPDSM, a unified big-data administration tool for Vertica and Hadoop at: HP DSM Download https://vertica.hpwsportal.com/#/Category/%7B%22categoryId%22%3A10185%7D/Show “People don’t know what they want until you show it to them… Our task is to read things that are not yet on the page.” — Steve Jobs From: Jain, Rohit (Trafodion) Sent: Wednesday, February 11, 2015 11:54 AM To: Nick Dimiduk; dev@phoenix.apache.org Cc: Tapper, Gunnar Subject: RE: Phoenix monitoring Thanks Nick! Yes Data Services Manager (DSM) does provide the kind of perspective Pari is looking for but not for Phoenix. Actually, if Phoenix is providing metrics and information in a repository or via a REST interface, on queries, elapsed times, etc. then certainly it could be incorporated into DSM. Trafodion is doing that now in R1.0 and we are working on having DSM provide full support for Trafodion along with complete support for HBase (some of it being already there), in order to provide a comprehensive view of all workloads from HDFS, HBase, to SQL. We did have plans to provide a SDK so that data services like Phoenix could be easily plugged into DSM. Not a hard task – just time and effort ☺. If Pari is interested, we can certainly demo the tool and go from there based on interest. Rohit From: Nick Dimiduk [mailto:ndimi...@gmail.com] Sent: Wednesday, February 11, 2015 9:01 AM To: dev@phoenix.apache.orgmailto:dev@phoenix.apache.org Cc: Jain, Rohit (Trafodion) Subject: Re: Phoenix monitoring Hi Pari, I'm not aware of a Phoenix-aware, end-to-end solution here. You can probably write a custom collector for a Phenox application that can report into OpenTSDB; I think JDBC metrics are available via JMX. That would be a great addition for that project! I've also seen a product demo from HP that does a lot of this for other tools on HBase, though I don't think it supports Phoenix yet (cc Rohit). Thanks, Nick On Wednesday, February 11, 2015, Pariksheet Barapatre pbarapa...@gmail.commailto:pbarapa...@gmail.com wrote: Hello All, Can you suggest good monitoring tool for phoenix. Our QA hbase cluster is crashing randomly. We wanted to know stats like which query causing the issue and how many queries running on that particular time window, resource consumption of query etc. Cheers Pari -- Cheers, Pari -- Cheers, Pari
