Re: Sequence number

[jesse]

Please see comments in line.

On Tue, Oct 22, 2019, 3:40 PM Josh Elser  wrote:

> Are you saying that you didn't restart the Phoenix QueryServer after you
> restored the Phoenix system tables?
>

We did not restart PQS after restoration of system tables.


> And then, after running into issues, you restarted PQS and then it
> worked as expected?
>

Yes, it fixed the issue.


> I can respect that we probably don't say this anywhere, but you should
> definitely be restarting any Phoenix clients (including PQS) if you are
> wiping the Phoenix system tables.
>


Got it, thanks.



> On 10/22/19 5:08 PM, jesse wrote:
> > It is properly restored, we double checked.
> >
> > We worked around the issue by restarting the query server.
> >
> > But it seems a bad bug.
> >
> >
> >
> >
> >
> > On Tue, Oct 22, 2019, 11:34 AM Thomas D'Silva  > <mailto:twdsi...@gmail.com>> wrote:
> >
> > Are you sure SYSTEM.SEQUENCE was restored properly? What is the
> > current value of the sequence in the restored table?
> >
> > On Fri, Oct 4, 2019 at 1:52 PM jesse  > <mailto:chat2je...@gmail.com>> wrote:
> >
> > Let's say there is a running cluster A, with table:books and
> > system.sequence current value 5000, cache size 100, incremental
> > is 1, the latest book with sequence id:4800
> >
> > Now the cluster A snapshot is backed up & restored into cluster
> > b, system.sequence and books table are properly restored, when
> > we add a new book, the book gets sequence id: 12, why it is not
> > 4801 or 5001?
> >
> > Our Phoenix version : 4.14.2
> >
> > Thanks
> >
> >
> >
>

Re: Sequence number

[jesse]

It is properly restored, we double checked.

We worked around the issue by restarting the query server.

But it seems a bad bug.





On Tue, Oct 22, 2019, 11:34 AM Thomas D'Silva  wrote:

> Are you sure SYSTEM.SEQUENCE was restored properly? What is the current
> value of the sequence in the restored table?
>
> On Fri, Oct 4, 2019 at 1:52 PM jesse  wrote:
>
>> Let's say there is a running cluster A, with table:books and
>> system.sequence current value 5000, cache size 100, incremental is 1, the
>> latest book with sequence id:4800
>>
>> Now the cluster A snapshot is backed up & restored into cluster b,
>> system.sequence and books table are properly restored, when we add a new
>> book, the book gets sequence id: 12, why it is not 4801 or 5001?
>>
>> Our Phoenix version : 4.14.2
>>
>> Thanks
>>
>>
>>
>>

Sequence number

[jesse]

Let's say there is a running cluster A, with table:books and
system.sequence current value 5000, cache size 100, incremental is 1, the
latest book with sequence id:4800

Now the cluster A snapshot is backed up & restored into cluster b,
system.sequence and books table are properly restored, when we add a new
book, the book gets sequence id: 12, why it is not 4801 or 5001?

Our Phoenix version : 4.14.2

Thanks

Re: What is the phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

[jesse]

Josh, in your sample project pom.xml file, the following build dependence
is not needed:


org.apache.phoenix
phoenix-server-client
4.7.0-HBase-1.1-SNAPSHOT


On Thu, Sep 19, 2019, 10:53 AM jesse  wrote:

> A) phoenix-4.14.2-HBase-1.4-thin-client.jar
>
> Just A) is good enough, Josh, you had a sample program here:
> https://github.com/joshelser/phoenix-queryserver-jdbc-client
>
> And the phoenix-4.14.2-HBase-1.4-thin-client.jar already contains the
> org.apache.phoenix.queryserver.client.Driver
>
>
>
>
>
>
> On Thu, Sep 19, 2019, 8:54 AM jesse  wrote:
>
>> You confused me more, if I write a Java program with http endpoint to PQS
>> for Phoenix read/write functions, should I depend on
>>
>> A) phoenix-4.14.2-HBase-1.4-thin-client.jar
>>
>> B) phoenix-queryserver-client-4.14.2-HBase-1.4.jar
>>
>> C) both
>>
>>
>>
>> On Thu, Sep 19, 2019, 4:12 AM Josh Elser  wrote:
>>
>>> "phoenix-queryserver-client" is the name of the Maven module which holds
>>> the required code for the "JDBC thin client", aka PQS client, aka
>>> "queryserver client".
>>>
>>> Maven convention is that a jar with the name of the Maven module is
>>> created.
>>>
>>> However, the majority of the code for the thin client is pulled from
>>> another Apache project. In fact, we only have one piece of code that we
>>> maintain client-side to connect to PQS.
>>>
>>> That third party code _does_ need to be included on the classpath for
>>> you to use the client. Thus, a shaded jar is created, with the
>>> human-readable name "thin-client" to make it very clear to you that this
>>> is the jar the use.
>>>
>>> The Maven build shows how all of this work.
>>>
>>> On 9/18/19 8:04 PM, jesse wrote:
>>> > It seems it is just the sqllinewrapper client, so confusing name...
>>> >
>>> >
>>> >
>>> > On Wed, Sep 18, 2019, 4:46 PM jesse >> > <mailto:chat2je...@gmail.com>> wrote:
>>> >
>>> > For query via PQS, we are using
>>> phoenix-4.14.2-HBase-1.4-thin-client.jar
>>> >
>>> > Then what is purpose and usage
>>> > of phoenix-queryserver-client-4.14.2-HBase-1.4.jar?
>>> >
>>> > Thanks
>>> >
>>>
>>

Re: What is the phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

[jesse]

A) phoenix-4.14.2-HBase-1.4-thin-client.jar

Just A) is good enough, Josh, you had a sample program here:
https://github.com/joshelser/phoenix-queryserver-jdbc-client

And the phoenix-4.14.2-HBase-1.4-thin-client.jar already contains the
org.apache.phoenix.queryserver.client.Driver






On Thu, Sep 19, 2019, 8:54 AM jesse  wrote:

> You confused me more, if I write a Java program with http endpoint to PQS
> for Phoenix read/write functions, should I depend on
>
> A) phoenix-4.14.2-HBase-1.4-thin-client.jar
>
> B) phoenix-queryserver-client-4.14.2-HBase-1.4.jar
>
> C) both
>
>
>
> On Thu, Sep 19, 2019, 4:12 AM Josh Elser  wrote:
>
>> "phoenix-queryserver-client" is the name of the Maven module which holds
>> the required code for the "JDBC thin client", aka PQS client, aka
>> "queryserver client".
>>
>> Maven convention is that a jar with the name of the Maven module is
>> created.
>>
>> However, the majority of the code for the thin client is pulled from
>> another Apache project. In fact, we only have one piece of code that we
>> maintain client-side to connect to PQS.
>>
>> That third party code _does_ need to be included on the classpath for
>> you to use the client. Thus, a shaded jar is created, with the
>> human-readable name "thin-client" to make it very clear to you that this
>> is the jar the use.
>>
>> The Maven build shows how all of this work.
>>
>> On 9/18/19 8:04 PM, jesse wrote:
>> > It seems it is just the sqllinewrapper client, so confusing name...
>> >
>> >
>> >
>> > On Wed, Sep 18, 2019, 4:46 PM jesse > > <mailto:chat2je...@gmail.com>> wrote:
>> >
>> > For query via PQS, we are using
>> phoenix-4.14.2-HBase-1.4-thin-client.jar
>> >
>> > Then what is purpose and usage
>> > of phoenix-queryserver-client-4.14.2-HBase-1.4.jar?
>> >
>> > Thanks
>> >
>>
>

Re: What is the phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

[jesse]

You confused me more, if I write a Java program with http endpoint to PQS
for Phoenix read/write functions, should I depend on

A) phoenix-4.14.2-HBase-1.4-thin-client.jar

B) phoenix-queryserver-client-4.14.2-HBase-1.4.jar

C) both



On Thu, Sep 19, 2019, 4:12 AM Josh Elser  wrote:

> "phoenix-queryserver-client" is the name of the Maven module which holds
> the required code for the "JDBC thin client", aka PQS client, aka
> "queryserver client".
>
> Maven convention is that a jar with the name of the Maven module is
> created.
>
> However, the majority of the code for the thin client is pulled from
> another Apache project. In fact, we only have one piece of code that we
> maintain client-side to connect to PQS.
>
> That third party code _does_ need to be included on the classpath for
> you to use the client. Thus, a shaded jar is created, with the
> human-readable name "thin-client" to make it very clear to you that this
> is the jar the use.
>
> The Maven build shows how all of this work.
>
> On 9/18/19 8:04 PM, jesse wrote:
> > It seems it is just the sqllinewrapper client, so confusing name...
> >
> >
> >
> > On Wed, Sep 18, 2019, 4:46 PM jesse  > <mailto:chat2je...@gmail.com>> wrote:
> >
> > For query via PQS, we are using
> phoenix-4.14.2-HBase-1.4-thin-client.jar
> >
> > Then what is purpose and usage
> > of phoenix-queryserver-client-4.14.2-HBase-1.4.jar?
> >
> > Thanks
> >
>

Re: What is the phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

[jesse]

It seems it is just the sqllinewrapper client, so confusing name...



On Wed, Sep 18, 2019, 4:46 PM jesse  wrote:

> For query via PQS, we are using phoenix-4.14.2-HBase-1.4-thin-client.jar
>
> Then what is purpose and usage
> of phoenix-queryserver-client-4.14.2-HBase-1.4.jar?
>
> Thanks
>

What is the phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

[jesse]

For query via PQS, we are using phoenix-4.14.2-HBase-1.4-thin-client.jar

Then what is purpose and usage
of phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

Thanks

Re: Buckets VS regions

[jesse]

Yes, that seems to be a trick, the issue is only associated a region.

What could be the causes of guidposts not updated?


On Mon, Aug 19, 2019, 6:40 PM venkata subbarayudu 
wrote:

> Did you try with updating table statistics, it may be because some times
> table guideposts are out of sync
>
> Below is the SQL to update table stats
>  Update statistics table
> By default above executes asynchronously, hence it may take some time to
> update depending on table size
>
> On Tue 20 Aug, 2019, 6:34 AM jesse,  wrote:
>
>> And the table is simple and has no index set up.
>>
>> On Mon, Aug 19, 2019, 6:03 PM jesse  wrote:
>>
>>> we got some trouble, maybe someone could shed some light on this.
>>>
>>> Table has primary key c1, c2 and c3.
>>> Table is set with SALT_BUCKETS=12. Now it has 14 regions.
>>>
>>> The table has a record with c1='a', c2='b', c3='c'
>>>
>>> If Phoenix query is like:
>>> select * from t where c2='b', it returns some results.
>>>
>>> select * from t where c1='a', it returns empty
>>> select * from t where c2='b' and c1='a', it returns empty
>>>
>>> select * from t where c3='c', it returns right results
>>> select * from t where c2='b' and c3='c', it returns results
>>>
>>> What the heck is going wrong? The system used to work fine.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Aug 19, 2019, 5:33 PM James Taylor 
>>> wrote:
>>>
>>>> It’ll start with 12 regions, but those regions may split as they’re
>>>> written to.
>>>>
>>>> On Mon, Aug 19, 2019 at 4:34 PM jesse  wrote:
>>>>
>>>>> I have a table is  SALT_BUCKETS = 12, but it has 14 regions, is this
>>>>> right?
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>

Re: Buckets VS regions

[jesse]

And the table is simple and has no index set up.

On Mon, Aug 19, 2019, 6:03 PM jesse  wrote:

> we got some trouble, maybe someone could shed some light on this.
>
> Table has primary key c1, c2 and c3.
> Table is set with SALT_BUCKETS=12. Now it has 14 regions.
>
> The table has a record with c1='a', c2='b', c3='c'
>
> If Phoenix query is like:
> select * from t where c2='b', it returns some results.
>
> select * from t where c1='a', it returns empty
> select * from t where c2='b' and c1='a', it returns empty
>
> select * from t where c3='c', it returns right results
> select * from t where c2='b' and c3='c', it returns results
>
> What the heck is going wrong? The system used to work fine.
>
>
>
>
>
>
>
>
> On Mon, Aug 19, 2019, 5:33 PM James Taylor  wrote:
>
>> It’ll start with 12 regions, but those regions may split as they’re
>> written to.
>>
>> On Mon, Aug 19, 2019 at 4:34 PM jesse  wrote:
>>
>>> I have a table is  SALT_BUCKETS = 12, but it has 14 regions, is this
>>> right?
>>>
>>> Thanks
>>>
>>>
>>>

Re: Buckets VS regions

[jesse]

we got some trouble, maybe someone could shed some light on this.

Table has primary key c1, c2 and c3.
Table is set with SALT_BUCKETS=12. Now it has 14 regions.

The table has a record with c1='a', c2='b', c3='c'

If Phoenix query is like:
select * from t where c2='b', it returns some results.

select * from t where c1='a', it returns empty
select * from t where c2='b' and c1='a', it returns empty

select * from t where c3='c', it returns right results
select * from t where c2='b' and c3='c', it returns results

What the heck is going wrong? The system used to work fine.








On Mon, Aug 19, 2019, 5:33 PM James Taylor  wrote:

> It’ll start with 12 regions, but those regions may split as they’re
> written to.
>
> On Mon, Aug 19, 2019 at 4:34 PM jesse  wrote:
>
>> I have a table is  SALT_BUCKETS = 12, but it has 14 regions, is this
>> right?
>>
>> Thanks
>>
>>
>>

Buckets VS regions

[jesse]

I have a table is  SALT_BUCKETS = 12, but it has 14 regions, is this right?

Thanks

Re: Phoenix with multiple HBase masters

[jesse]

After further investigation, I figured there are two types of clients: thin
client and thick client.

- thin client talks to PQS, only ELB or PQS http address is supported
- thick client supports ZK

I don't know the PQS heath url yet, not documented anywhere. Any one knows?



On Wed, Aug 7, 2019, 12:06 PM Aleksandr Saraseka 
wrote:

> I didn't try this personally, but according to the documentation
> https://phoenix.apache.org/server.html it has possibility to have
> ZK-based load-balancing. Please refer to the "Load balancing" section at
> the bottom.
>
> On Wed, Aug 7, 2019 at 9:14 PM jesse  wrote:
>
>> Thank you all, very helpful information.
>>
>> 1) for server side ELB, what is the PQS health check url path?
>>
>> 2) Does Phoenix client library support client-side load-balancing? i. e
>> client gets list of PQS addresses from ZK, and performs load balancing.
>> Thus ELB won't be needed.
>>
>>
>>
>> On Wed, Aug 7, 2019, 9:01 AM Josh Elser  wrote:
>>
>>> Great answer, Aleksandr!
>>>
>>> Also worth mentioning there is only ever one active HBase Master at a
>>> time. If you have multiple started, one will be active as the master and
>>> the rest will be waiting as a standby in case the current active master
>>> dies for some reason (expectedly or unexpectedly).
>>>
>>> On 8/7/19 9:55 AM, Aleksandr Saraseka wrote:
>>> > Hello.
>>> > - Phoenix libs should be installed only on RegionServers.
>>> > - QueryServer - it's up to you, PQS can be installed anywhere you want
>>> > - No. QueryServer is using ZK quorum to get everything it needs
>>> > - If you need to balance traffic with multiply PQSs - then yes, but
>>> > again - it's up to you. It is not required multiply PQSs if you have
>>> > multiply HBase masters.
>>> >
>>> > On Wed, Aug 7, 2019 at 12:58 AM jesse >> > <mailto:chat2je...@gmail.com>> wrote:
>>> >
>>> > Our cluster used to have one hbase master, now a secondary is
>>> added.
>>> > For phonenix, what changes should we make?
>>> >   - do we have to install new hbase libraries on the new hbase
>>> > master node?
>>> > - do we need to install new query server on the hbase master?
>>> > - any configuration changes should we make?
>>> > - do we need an ELB for the query server?
>>> >
>>> > Thanks
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >   Aleksandr Saraseka
>>> > DBA
>>> > 380997600401
>>> >  *•* asaras...@eztexting.com
>>> > <mailto:asaras...@eztexting.com> *•* eztexting.com
>>> > <
>>> http://eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> >
>>> >
>>> > <
>>> http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> > <
>>> http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> > <
>>> http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> > <
>>> https://www.youtube.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> > <
>>> https://www.instagram.com/ez_texting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> > <
>>> https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> > <
>>> https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
>>> >
>>> >
>>>
>>
>
> --
> Aleksandr Saraseka
> DBA
> 380997600401
>  *•*  asaras...@eztexting.com  *•*  eztexting.com
> <http://eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> <http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.youtube.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.instagram.com/ez_texting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>

Re: Phoenix with multiple HBase masters

[jesse]

Thank you all, very helpful information.

1) for server side ELB, what is the PQS health check url path?

2) Does Phoenix client library support client-side load-balancing? i. e
client gets list of PQS addresses from ZK, and performs load balancing.
Thus ELB won't be needed.



On Wed, Aug 7, 2019, 9:01 AM Josh Elser  wrote:

> Great answer, Aleksandr!
>
> Also worth mentioning there is only ever one active HBase Master at a
> time. If you have multiple started, one will be active as the master and
> the rest will be waiting as a standby in case the current active master
> dies for some reason (expectedly or unexpectedly).
>
> On 8/7/19 9:55 AM, Aleksandr Saraseka wrote:
> > Hello.
> > - Phoenix libs should be installed only on RegionServers.
> > - QueryServer - it's up to you, PQS can be installed anywhere you want
> > - No. QueryServer is using ZK quorum to get everything it needs
> > - If you need to balance traffic with multiply PQSs - then yes, but
> > again - it's up to you. It is not required multiply PQSs if you have
> > multiply HBase masters.
> >
> > On Wed, Aug 7, 2019 at 12:58 AM jesse  > <mailto:chat2je...@gmail.com>> wrote:
> >
> > Our cluster used to have one hbase master, now a secondary is added.
> > For phonenix, what changes should we make?
> >   - do we have to install new hbase libraries on the new hbase
> > master node?
> > - do we need to install new query server on the hbase master?
> > - any configuration changes should we make?
> > - do we need an ELB for the query server?
> >
> > Thanks
> >
> >
> >
> >
> >
> > --
> >   Aleksandr Saraseka
> > DBA
> > 380997600401
> >  *•* asaras...@eztexting.com
> > <mailto:asaras...@eztexting.com> *•* eztexting.com
> > <
> http://eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> >
> >
> > <
> http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> https://www.youtube.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> https://www.instagram.com/ez_texting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
> >
> >
>

Phoenix with multiple HBase masters

[jesse]

Our cluster used to have one hbase master, now a secondary is added. For
phonenix, what changes should we make?
 - do we have to install new hbase libraries on the new hbase master node?
- do we need to install new query server on the hbase master?
- any configuration changes should we make?
- do we need an ELB for the query server?

Thanks

Re: is Apache phoenix reliable enough?

[jesse]

Have you considered : CockRoach DB?

It is similar to Google Spanner but with soft clock mechanism.







On Sun, Jun 23, 2019, 11:02 PM Hengesbach, Martin <
martin.hengesb...@fiz-karlsruhe.de> wrote:

> Hi,
>
>
>
> we are using Phoenix in production since more than 2 years. We are quite
> unhappy with the reliability of Phoenix. We migrated from Oracle because of
> the performance (we have tables with up to 200M records, each record up to
> 30 MB in size, up to 10 selective columns). Phoenix is really much faster
> than Oracle (20 nodes cluster).
>
>
>
> But we have problems with
>
> · Phoenix totally hanging, only restart helps (We were able to
> reduce this from daily to monthly)
>
> · incorrect indices, need rebuild
>
> · select statements not producing the expected (specified) results
>
> · Upserts sporadically not working without error message
>
> · Some not reproducible errors
>
> · …
>
>
>
> We are thinking about switching to another database, but the question is:
> what is better (reliable and performant)?
>
>
>
> Conclusion: With the current status of Phoenix, I would never use it again.
>
>
>
> Regards
>
> Martin
>
>
>
>
>
>
>
> *Von:* jesse [mailto:chat2je...@gmail.com]
> *Gesendet:* Samstag, 22. Juni 2019 20:04
> *An:* user@phoenix.apache.org
> *Betreff:* is Apache phoenix reliable enough?
>
>
>
> I stumbled on this post:
>
>
> https://medium.com/@vkrava4/fighting-with-apache-phoenix-secondary-indexes-163495bcb361
>
>
> and the bug:
>
>https://issues.apache.org/jira/browse/PHOENIX-5287
>
>
>  I had a similar very frustrating experience with Phoenix, In addition to
> various performance issues, you can found one of my posts about the
> reliability issue on the mail-list.
>
>
>
>
> https://lists.apache.org/thread.html/231b175fce8811d474cceb1fe270a3dd6b30c9eff2150ac42ddef0dc@%3Cuser.phoenix.apache.org%3E
>
>
>
>  just wondering others experience if you could share
>
>
>
>
>
>
>
>
>
>
> --
>
> FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
> Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB
> 101892.
> Geschäftsführerin: Sabine Brünger-Weilandt.
> Vorsitzende des Aufsichtsrats: MinDirig'in Dr. Angelika Willms-Herget.
> FIZ Karlsruhe ist zertifiziert mit dem Siegel "audit berufundfamilie".
>

is Apache phoenix reliable enough?

[jesse]

I stumbled on this post:


https://medium.com/@vkrava4/fighting-with-apache-phoenix-secondary-indexes-163495bcb361

and the bug:

   https://issues.apache.org/jira/browse/PHOENIX-5287

 I had a similar very frustrating experience with Phoenix, In addition to
various performance issues, you can found one of my posts about the
reliability issue on the mail-list.

https://lists.apache.org/thread.html/231b175fce8811d474cceb1fe270a3dd6b30c9eff2150ac42ddef0dc@%3Cuser.phoenix.apache.org%3E

 just wondering others experience if you could share

Is 200k data in a column a big concern?

[jesse]

It seems the write take a long time and the system substantially slows down
with requests.

however, hbase official doc mentions soft limit is 32mb.

Re: A strange question about Phoenix

[jesse]

I manually did update system.stat. No help.

We are running amazon EMR.



On Thu, Jun 20, 2019, 2:29 PM Josh Elser  wrote:

> Make sure you have updated statistics for your table. Depending on the
> last time you created the stats, you may have to manually delete the
> stats from SYSTEM.STATS (as there are safeguards to prevent re-creating
> statistics too frequently).
>
> There have been some bugs in the past that results from invalid stats
> guideposts.
>
> On 6/19/19 3:25 PM, jesse wrote:
> > 1) hbase clone-snapshot into my_table
> > 2) sqlline.py zk:port console  to create my_table.
> >
> > Very straight forward.
> >
> >
> >
> > On Wed, Jun 19, 2019, 11:40 AM anil gupta  > <mailto:anilgupt...@gmail.com>> wrote:
> >
> > Sounds strange.
> > What steps you followed to restore snapshot of Phoenix table?
> >
> > On Tue, Jun 18, 2019 at 9:34 PM jesse  > <mailto:chat2je...@gmail.com>> wrote:
> >
> > hi:
> >
> >   When my table is restored via hbase clone-snapshot,
> >
> > 1) sqlline.py console shows the proper number of records:
> > select count(*) from my_table.
> > 2) select my_column from my_table limit 1 works fine.
> >
> >   However, select * from my_table limit 1; returns no row.
> >
> >   Do I need to perform some extra operations?
> >
> >   thanks
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>

Re: A strange question about Phoenix

[jesse]

1) hbase clone-snapshot into my_table
2) sqlline.py zk:port console  to create my_table.

Very straight forward.



On Wed, Jun 19, 2019, 11:40 AM anil gupta  wrote:

> Sounds strange.
> What steps you followed to restore snapshot of Phoenix table?
>
> On Tue, Jun 18, 2019 at 9:34 PM jesse  wrote:
>
>> hi:
>>
>>  When my table is restored via hbase clone-snapshot,
>>
>> 1) sqlline.py console shows the proper number of records:  select
>> count(*) from my_table.
>> 2) select my_column from my_table limit 1 works fine.
>>
>>  However, select * from my_table limit 1; returns no row.
>>
>>  Do I need to perform some extra operations?
>>
>>  thanks
>>
>>
>>
>>
>>
>>
>>
>
> --
> Thanks & Regards,
> Anil Gupta
>

A strange question about Phoenix

[jesse]

hi:

 When my table is restored via hbase clone-snapshot,

1) sqlline.py console shows the proper number of records:  select count(*)
from my_table.
2) select my_column from my_table limit 1 works fine.

 However, select * from my_table limit 1; returns no row.

 Do I need to perform some extra operations?

 thanks

Re: Phoenix and HBase Snapshots

[Jesse Yates]

Just have to make sure you don't have schema change during snapshots

On Fri, Feb 12, 2016 at 6:24 PM Gaurav Agarwal 
wrote:

> We can take snapshot or restore snapshot from hbase of the phoenix tables.
> Export/import feature also hbase provide to us.
>
> Thanks
> On Feb 13, 2016 3:15 AM, "Nick Dimiduk"  wrote:
>
>> Heya,
>>
>> Has anyone tried taking/restoring snapshots of Phoenix tables? How does
>> that work out? Presumably we would require a sibling snapshot of some
>> system tables as well. Have you tried the Export/Import/Restore workflow?
>> Have you tried MR or Spark over the snapshots?
>>
>> Curious,
>> -n
>>
>

Re: Pheonix on standalone HBase for production

[Jesse Yates]

Yes, lots of ppl do, including folks at Salesforce. You need to setup your
own query tuning infra to make sure it runs ok

On Tue, Jan 26, 2016, 8:26 AM John Lilley  wrote:

> Does anyone ever use Phoenix on standalone Hbase for production?  Is it
> advisable?
>
>
>
> *John Lilley*
>
>
>

Re: Phoenix Query Server Placement

[Jesse Yates]

I think he means its not a terribly expensive process - it is basically
just a fancy query proxy. If you are running a cluster any larger than 3
nodes you should seriously consider running at least a second or third
HMaster. When they are in standby mode they don't do very much - just watch
ZK for the leader lock. When active they very very rarely impact the path
of the client and you should be able to pretty easily ride over a hmaster
failover event.

However, given that it is a pretty new component, I would personally be
cautious putting it on nodes that affect the entire cluster health, like
the servers hosting ZK or the NN.

Just my $0.02

On Sat, Dec 5, 2015 at 4:20 AM Andreas Fritzler 
wrote:

> Thanks Thomas. Yes, that's the one.
>
> What exactly do you mean by "heavy"? I was wondering if it would make
> sense to put it on the same host as the HBase master. On the other side I
> don't want the query server bring down the master machine ...
>
>
> On Fri, Dec 4, 2015 at 4:56 PM, Thomas Decaux  wrote:
>
>> You mean https://phoenix.apache.org/server.html?
>>
>> I am not sure it's really heavy, so I would suggest to run it closest
>> from HBase server.
>>
>> 2015-12-04 15:59 GMT+01:00 Andreas Fritzler :
>>
>>> Hi,
>>>
>>> What is the best place to install the Phoenix Query Server within an HDP
>>> Hadoop cluster? Is it a good idea to place it on one of the master
>>> machines? Or would your rather suggest to have a dedicated host for that?
>>>
>>> Also any suggestion/best practices on how to load balance multiple
>>> instances of the query server?
>>>
>>> Thanks a lot in advance!
>>>
>>> Regards,
>>> Andreas
>>>
>>
>>
>

Re: Enabling secondary index in HBase - Phoenix

[Jesse Yates]

I think with that version of Phoenix you should have that class.

1. Can you grep the jar contents and ensure the class (IndexedWALEditCodec)
is there?
2. Can you check the hbase classpath to ensure the jar is getting picked
up? (bin/hbase classpath)

On Sat, Nov 28, 2015, 6:10 PM Saba Varatharajaperumal 
wrote:

> Hi
>
> I have a 3 node hbase (version 1.1.2) cluster on which I am trying to use
> Phoenix.  I am using Phoenix version 4.5.2. I have copied
> "phoenix-4.5.2-HBase-1.1-server.jar"  to hbase/lib folder and changed the
> hbase-sites.xml as described in the phoenix web site. When I start the
> hbase server, I get the below error,
>
> The phoenix-4.5.2-HBase-1.1-server.jar file definitely under hbase lib
> folder on all servers including master.
>
> Can someone help me on this?
>
> java.io.IOException: Cannot get log reader
> at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:344)
> at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:266)
> at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:839)
> at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:763)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: Unable to find
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:36)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:103)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:280)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:290)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:147)
> at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:301)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:32)
> ... 17 more
>
>
> Thanks and appreciate your time
>
> Saba
>
>

Re: blog describing new time-series data optimization

[Jesse Yates]

Great post, awesome to see the optimization going in.

Would be cool to see if we could roll in some of the stuff talked about at
the last meetup too :)

On Sun, Nov 8, 2015, 11:27 AM James Taylor  wrote:

> Thanks, Juan. I fixed the typo.
>
> On Sun, Nov 8, 2015 at 11:21 AM, Samarth Jain 
> wrote:
>
>> Glad you find it useful, Juan and thanks for reporting the typo. The
>> column name should be EVENT_DATE. We will get it changed.
>>
>> On Sun, Nov 8, 2015 at 11:18 AM, Juan Rodríguez Hortalá <
>> juan.rodriguez.hort...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Very interesting post and very useful feature. Just a minor remark, I
>>> think there is a typo in the post, in the CREATE TABLE EVENTS_RTS statement
>>>
>>> CREATE TABLE *EVENTS_RTS* (
>>> EVENT_ID CHAR(15) NOT NULL,
>>> EVENT_TYPE CHAR(3) NOT NULL,
>>> EVENT_TIME DATE NOT NULL,
>>> APPLICATION_TYPE VARCHAR,
>>> SOURCE_IP VARCHAR
>>> CONSTRAINT PK PRIMARY KEY (
>>> EVENT_ID,
>>> EVENT_TYPE,
>>> EVENT_DATE *ROW_TIMESTAMP*))
>>>
>>> I think the column EVENT_TIME should be EVENT_DATE. Or maybe I'm not
>>> understanding this correctly.
>>>
>>> Greetings,
>>>
>>> Juan
>>>
>>>
>>>
>>> On Sat, Nov 7, 2015 at 6:53 PM, James Taylor 
>>> wrote:
>>>
 If you have time-series data for which you'd like to improve query
 performance, take a look at this[1] blog written by Samarth Jain on a new
 feature in our 4.6 release:

 https://blogs.apache.org/phoenix/entry/new_optimization_for_time_series

 Enjoy!

 James

>>>
>>>
>>
>

Re: ResultSet size

[Jesse Yates]

Correct. So you have to make sure that you have enough memory to handle the
fetchSize * concurrent requests.


On Tue, Oct 6, 2015 at 10:34 AM Sumit Nigam  wrote:

> Thanks Samarth and Jesse.
>
> So, in effect setting the batch size (say, stmt.setFetchSize()) ensures
> that only that much data is copied over the wire en-mass? And
> 'behind-the-scenes', Phoenix driver would fetch next batch as each fetch
> size is exhausted?
>
> Thanks,
> Sumit
>
> --
> *From:* Samarth Jain 
> *To:* "user@phoenix.apache.org" 
> *Cc:* Sumit Nigam 
> *Sent:* Tuesday, October 6, 2015 9:20 PM
> *Subject:* Re: ResultSet size
>
> To add to what Jesse said, you can override the default scanner fetch size
> programmatically via Phoenix by calling statement.setFetchSize(int).
> On Tuesday, October 6, 2015, Jesse Yates  wrote:
>
>
> So HBase (and by extension, Phoenix) does not do true "streaming" of rows
> - rows are copied into memory from the HFiles and then eventually copied
> en-mass onto the wire. On the client they are pulled off in chunks and
> paged through by the client scanner. You can control the batch size (amount
> of rows in each 'page') via the usual HBase client configurations
>
> On Tue, Oct 6, 2015 at 8:38 AM Sumit Nigam  wrote:
>
> Hi,
>
> Does Phoenix buffer the result set internally? I mean when I fire a huge
> skip scan IN clause, then the data being returned may be too huge to
> contain in memory. So, ideally I'd like to stream data through the
> resultset.next() method. So, my question is does Phoenix really stream
> results?
>
> And if so, is there a way to control how much is loads in one time in
> client side before its next() fetches next batch of data from region
> servers to client?
>
> Best regards,
> Sumit
>
>
>
>

Re: ResultSet size

[Jesse Yates]

So HBase (and by extension, Phoenix) does not do true "streaming" of rows -
rows are copied into memory from the HFiles and then eventually copied
en-mass onto the wire. On the client they are pulled off in chunks and
paged through by the client scanner. You can control the batch size (amount
of rows in each 'page') via the usual HBase client configurations

On Tue, Oct 6, 2015 at 8:38 AM Sumit Nigam  wrote:

> Hi,
>
> Does Phoenix buffer the result set internally? I mean when I fire a huge
> skip scan IN clause, then the data being returned may be too huge to
> contain in memory. So, ideally I'd like to stream data through the
> resultset.next() method. So, my question is does Phoenix really stream
> results?
>
> And if so, is there a way to control how much is loads in one time in
> client side before its next() fetches next batch of data from region
> servers to client?
>
> Best regards,
> Sumit
>

Re: [DISCUSS] discontinue 4-x HBase 1.0 releases?

[Jesse Yates]

Along the same lines as HBase, it seems fine to discontinue old code lines
until such point as there is someone willing to maintain a given line - a
new RM. Essentially, its the same as an RM stepping down from managing a
release and no releases happening until someone cares enough to make a new
one.

That said, if there are any users who want to step up and help maintain a
release that is no longer managed (ie. b/c you are running an older version
and can't upgrade) that would be fantastic.

On Fri, Sep 18, 2015 at 5:53 PM James Taylor  wrote:

> +user list
>
> Please let us know if you're counting on HBase 1.0 support given that we
> have HBase 1.1 support.
>
> Thanks,
> James
>
> On Fri, Sep 18, 2015 at 4:11 PM, Nick Dimiduk  wrote:
>
> > Seems like a better question for the users mailing list. Probably it
> > depends on where the vendors are going, if there will be a long-running
> > 1.0.x release line, we should keep it around. From Apache perspective,
> it's
> > probably fine to drop 1.0.x in favor of 1.1.x. HBase 1.2 is right around
> > the corner too...
> >
> > On Thu, Sep 17, 2015 at 4:42 PM, James Taylor 
> > wrote:
> >
> > > Given that we have Phoenix support for HBase 1.1, does anyone feel like
> > we
> > > need to continue with Phoenix releases for HBase 1.0?
> > >
> > > Thanks,
> > > James
> > >
> >
>

Re: Can we create an secondary index for two or more columns?

[Jesse Yates]

>
> create index my_idx on EXAMPLE (M.C0, M.C1)


This will create an index on both columns _at the same time_. This means
the row key in the index will be a combination of both columns. Creating an
index like this is only an advantage if you are commonly querying _both
columns_ at the same time. For instance, a query like:

Select * from EXAMPLE WHERE m.c0 = "a" AND m.c1 = "b" will leverage the
index on both columns. However, if you are just querying each column
separately, then using your solution (b) will be better.

Does that make sense?

---
Jesse Yates
@jesse_yates
jyates.github.com

On Sun, May 17, 2015 at 6:17 PM, Tao Xiao  wrote:

> Thanks to Yuhao,
>
> Is the column *M.C1* ignored when I invoke " create index idx on EXAMPLE
> (M.C0, M.C1) " ?
>
> Which of the following two solutions make sense?
>
> solution a):
>  create index idx_1 on EXAMPLE (M.C0, M.C1)
>  create index idx_2 on EXAMPLE (M.C1, M.C0)
>
>
> solution b):
> create index idx_1 on EXAMPLE (M.C0)
> create index idx_2 on EXAMPLE (M.C1)
>
>
>
>
> 2015-05-17 22:34 GMT+08:00 Yuhao Bi :
>
>> Hi Xiao Tao,
>>
>> You can create another secondary index on the same columns.
>> create index another_index on EXAMPLE(M.C1, M.C0)
>>
>> After doing that, query b) should return faster.
>>
>> 2015-05-17 22:25 GMT+08:00 Tao Xiao :
>>
>>> I tried to build secondary index for two columns, *M.C0* and *M.C1*:
>>>
>>>   create index my_idx on EXAMPLE (M.C0, M.C1)
>>>
>>>
>>> Then I tried to query by two indexes, respectively:
>>>
>>> a). select M.C0 from EXAMPLE where M.C0 = 'c0_0001'
>>> b). select M.C1 from EXAMPLE where M.C1 = 'c1_0001'
>>>
>>> Query a) returned results in less than 0.3 second, and query b) returned
>>> results in about 13 seconds. It seems that index was built for column
>>> *M.C0*, not *M.C1*.
>>>
>>> Can we build secondary index for two or more columns ?
>>>
>>> Thanks.
>>>
>>
>>
>

Re: understanding phoenix code flow

[Jesse Yates]

You could understand it just by reading the code, or running the tests, or
running a HBase minicluster in the JVM or in standalone mode or in
pseudo-distributed mode or in a fully distributed setup.

What are you trying to achieve?

In the area you are interested in, have you:
 - read the docs
 - read the code
 - read though unit tests
 - debugged the unit tests and stepped through areas of uncertainty
 - written a unit test to figure out if your conjectures are correct

If you are still unclear, then we can gladly help.

On Fri, Apr 3, 2015 at 2:46 PM ashish tapdiya 
wrote:

> To understand phoenix code flow should HBase be steup in standalone or
> pseudo-distributed mode
>
> Thanks,
> ~Ashish
>

Re: Re: Phoenix tracing

[Jesse Yates]

It looks like you are using two different metrics files on the classpath of
the server. You can only have one (quirk of Hadoop's metrics2 system). The
configurations for the phoenix sinks should be in the
hadoop-metrics2-hbase.properties
file since HBase will load the metrics system before the phoenix CPs have a
chance to initialize it.

See "hadoop-metrics2-hbase.properties" under
http://phoenix.apache.org/tracing.html for what should go into the file for
phoenix tracing to be enabled.

On Fri, Apr 3, 2015 at 12:43 PM ashish tapdiya 
wrote:

> Hi,
>
> In one slave node's region-server logs i see
>
> 2015-04-03 14:07:47,162 INFO org.apache.phoenix.metrics.Metrics:
> Initializing metrics system: phoenix
> 2015-04-03 14:07:47,162 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system
> already initialized!
> 2015-04-03 14:07:47,163 WARN org.apache.phoenix.metrics.Metrics: Phoenix
> metrics2/tracing sink was not started. Should be it be?
>
> On Master and other slaves i dont see phoenix metrics but only hbase
> metrics initialization
>
> 2015-04-03 14:07:22,275 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2015-04-03 14:07:22,338 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2015-04-03 14:07:22,339 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system
> started
>
> Thanks,
> ~Ashish
>
>
> On Thu, Apr 2, 2015 at 8:43 PM, Fulin Sun  wrote:
>
>> Hi, there
>>
>> In your regionserver side logs, can you find something related with
>> phoenix tracing ?
>> If not, I assume you still had configs wrong placed.
>>
>> Thanks,
>> Sun.
>>
>> --
>> --
>>
>> CertusNet
>>
>>
>> *From:* ashish tapdiya 
>> *Date:* 2015-04-03 09:38
>> *To:* user 
>> *Subject:* Re: Phoenix tracing
>> Hi Sun,
>>
>> hbase-site.xml is already in bin directory and the directory is on
>> classpath
>>
>> Thanks,
>> ~Ashish
>>
>> On Thu, Apr 2, 2015 at 8:13 PM, Fulin Sun  wrote:
>>
>>> Hi,
>>>
>>> Maybe you want to put your client side hbase-site.xml into phoenix bin
>>> directory.
>>>
>>> I once encountered such problem and doing that just resolved it.
>>>
>>> Thanks,
>>> Sun.
>>>
>>> --
>>> --
>>>
>>> CertusNet
>>>
>>>
>>> *From:* ashish tapdiya 
>>> *Date:* 2015-04-03 04:53
>>> *To:* user 
>>> *Subject:* Phoenix tracing
>>> Hi,
>>>
>>> I am trying to setup tracing following the steps specified on the
>>> Phoenix webpage,
>>> http://phoenix.apache.org/tracing.html
>>>
>>> Below are the steps I have followed,
>>>
>>> 1. On the client, property files (hadoop-metrics2-phoenix.properties and
>>> hadoop-metrics2-hbase.properties) came configured  with the distribution
>>> and are on the classpath.
>>>
>>> 2. On the server nodes (master + slaves), appended following to the
>>> /usr/lib/hbase/conf/hadoop-metrics2-hbase.properties file,
>>>
>>> # ensure that we receive traces on the server
>>> hbase.sink.tracing.class=org.apache.phoenix.trace.PhoenixMetricsSink
>>> # Tell the sink where to write the metrics
>>> hbase.sink.tracing.writer-class=org.apache.phoenix.trace.PhoenixTableMetricsWriter
>>> # Only handle traces with a context of "tracing"
>>> hbase.sink.tracing.context=tracing
>>>
>>> 3. Added the following property to the client's hbase-site.xml
>>> 
>>>   
>>> phoenix.trace.frequency
>>> always
>>>   
>>> 
>>>
>>> 4. Created the SYSTEM.TRACING_STATS table using,
>>>
>>> CREATE TABLE *SYSTEM.TRACING_STATS* (
>>>   *trace_id* BIGINT NOT NULL,
>>>   *parent_id* BIGINT NOT NULL,
>>>   *span_id* BIGINT NOT NULL,
>>>   *description* VARCHAR,
>>>   *start_time* BIGINT,
>>>   *end_time* BIGINT,
>>>   *hostname* VARCHAR,
>>>   *tags.count* SMALLINT,
>>>   *annotations.count* SMALLINT,
>>>   CONSTRAINT pk PRIMARY KEY (*trace_id, parent_id, span_id*)
>>>
>>> 5. Restarted cluster
>>>
>>> However, after running queries from the client, SYSTEM.TRACING_STATS table 
>>> returns 0 rows.
>>>
>>>
>>> I am using cloudera distribution 5.3.2 with
>>>
>>> Hadoop  Ver: 2.5.0
>>> HBase Ver: 0.98.6
>>>
>>> Phoenix Ver: 4.3.0
>>>
>>> Thanks,
>>> ~Ashish
>>>
>>>
>>>
>>
>

Re: Select dynamic column content

[Jesse Yates]

And it looks like you already figured that out :)

On Tue, Jan 6, 2015, 9:43 AM Jesse Yates  wrote:

> You wouldn't even need another table, just a single VARCHAR[] column to
> keep the column names. Its ideal to keep it in the same row (possibly in
> another cf if you expect it to be large) since you get ACID compliance on
> that row, which you wouldn't get from using another table. You then just
> upsert the names column at the same time you upsert the dynamic columns.
>
> Phoenix does something similar for tracing where there is an unknown
> number of annotations - we keep a trace annotation count column which then
> can be used to figure out the dynamic annotation column names (which are
> things like annotations.a0, .a1, .a2, etc)
>
> The downside is that you then need to do two queries to get all the
> columns, but until we implement the cf.* logic for dynamic columns, that's
> the best you can do.
>
> - jesse
>
> On Tue, Jan 6, 2015, 9:23 AM Sumanta Gh  wrote:
>
>> Thanks Nicolas for replying.
>> I am already managing dynamic column names either putting them in a
>> separate column or keeping the names in cache.
>> But looking at the pace Phoenix is evolving, IMHO this cf.* query pattern
>> would be very much helpful for users.
>> The stock HBase client is capable of doing that.
>> Because my table is extremely sparsed and I allow quite a random schema
>> in every row of the table, getting the content of the dynamic column in a
>> single query will save lot of time.
>> Looking forward to your completion of the work...
>>
>> Regards
>> Sumanta
>>
>>
>> -Nicolas Maillard  wrote: -
>> To: user@phoenix.apache.org
>> From: Nicolas Maillard 
>> Date: 01/06/2015 03:08PM
>> Subject: Re: Select dynamic column content
>>
>>
>> Hello Sumanta
>>
>> This is a last bit missing in the select cf.* query pattern that would
>> bring back not only known columns but all dynamic ones also. I never got
>> around to finishing that bit for different reasons. The best way would be
>> to finish that, other than that I do not see an easy way to retrieve
>>  dynamic columns of which you have lost the column name. I guess if there
>> is a logic ti these column names you could try to reconstruct or keep a
>> second table of keys to dynamic column names to find them in the after math.
>>
>> regards
>>
>> On Tue, Jan 6, 2015 at 8:01 AM, Sumanta Gh  wrote:
>>
>>> Hi,
>>> It has been a puzzle for me to get the content of dynamic columns in a
>>> single SELECT * FROM query.
>>> Challenge is that I do not know the dynamic column names in advance so I
>>> can not mention a dynamic column in the SELECT query.
>>> Is there any way out? Please suggest.
>>>
>>> Regards
>>> Sumanta
>>>
>>> =-=-=
>>> Notice: The information contained in this e-mail
>>> message and/or attachments to it may contain
>>> confidential or privileged information. If you are
>>> not the intended recipient, any dissemination, use,
>>> review, distribution, printing or copying of the
>>> information contained in this e-mail message
>>> and/or attachments to it are strictly prohibited. If
>>> you have received this communication in error,
>>> please notify us by reply e-mail or telephone and
>>> immediately and permanently delete the message
>>> and any attachments. Thank you
>>>
>>>
>>
>>
>> --
>> Nicolas Maillard Solution Engineer
>> Phone: +33 (0)6 68 17 66 05
>> Email:  nmaillard*@hortonworks.com* 
>> Website:   http://www.hortonworks.com
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>

Re: Select dynamic column content

[Jesse Yates]

You wouldn't even need another table, just a single VARCHAR[] column to
keep the column names. Its ideal to keep it in the same row (possibly in
another cf if you expect it to be large) since you get ACID compliance on
that row, which you wouldn't get from using another table. You then just
upsert the names column at the same time you upsert the dynamic columns.

Phoenix does something similar for tracing where there is an unknown number
of annotations - we keep a trace annotation count column which then can be
used to figure out the dynamic annotation column names (which are things
like annotations.a0, .a1, .a2, etc)

The downside is that you then need to do two queries to get all the
columns, but until we implement the cf.* logic for dynamic columns, that's
the best you can do.

- jesse

On Tue, Jan 6, 2015, 9:23 AM Sumanta Gh  wrote:

> Thanks Nicolas for replying.
> I am already managing dynamic column names either putting them in a
> separate column or keeping the names in cache.
> But looking at the pace Phoenix is evolving, IMHO this cf.* query pattern
> would be very much helpful for users.
> The stock HBase client is capable of doing that.
> Because my table is extremely sparsed and I allow quite a random schema in
> every row of the table, getting the content of the dynamic column in a
> single query will save lot of time.
> Looking forward to your completion of the work...
>
> Regards
> Sumanta
>
>
> -Nicolas Maillard  wrote: -
> To: user@phoenix.apache.org
> From: Nicolas Maillard 
> Date: 01/06/2015 03:08PM
> Subject: Re: Select dynamic column content
>
>
> Hello Sumanta
>
> This is a last bit missing in the select cf.* query pattern that would
> bring back not only known columns but all dynamic ones also. I never got
> around to finishing that bit for different reasons. The best way would be
> to finish that, other than that I do not see an easy way to retrieve
>  dynamic columns of which you have lost the column name. I guess if there
> is a logic ti these column names you could try to reconstruct or keep a
> second table of keys to dynamic column names to find them in the after math.
>
> regards
>
> On Tue, Jan 6, 2015 at 8:01 AM, Sumanta Gh  wrote:
>
>> Hi,
>> It has been a puzzle for me to get the content of dynamic columns in a
>> single SELECT * FROM query.
>> Challenge is that I do not know the dynamic column names in advance so I
>> can not mention a dynamic column in the SELECT query.
>> Is there any way out? Please suggest.
>>
>> Regards
>> Sumanta
>>
>> =-=-=
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>>
>>
>
>
> --
> Nicolas Maillard Solution Engineer
> Phone: +33 (0)6 68 17 66 05
> Email:  nmaillard*@hortonworks.com* 
> Website:   http://www.hortonworks.com
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Secondary Indexes

[Jesse Yates]

The phoenix indexes are only kept up to date when writes are made through
the phoenix client.

A more "out there" option would be to write your own indexer plugin (the
actual name escapes me right now) that does similar writes when the Phoenix
plugin wouldn't do an update (eg. Non-phoenix client writes). However, that
is a fairly large undertaking - really only worth it if you need more
consistent indexes than as of every rebuild.

On Tue, Dec 30, 2014, 6:03 AM Siddharth Ubale 
wrote:

>  Hi All,
>
>
>
> I wanted to know that if I am using secondary indexes i.e Mutable indexes
> on my Phoenix tables, will indexes on them be updated whenever I enter
> values into the hbase table for which index is created or do I have to
> Alter –Rebuild the index everytime?
>
>
>
>
>
> Siddharth Ubale,
>
> *Synchronized Communications *
>
> *#43, Velankani Tech Park, Block No. II, *
>
> *3rd Floor, Electronic City Phase I,*
>
> *Bangalore – 560 100*
>
> *Tel : +91 80 3202 4060*
>
> *Web:* *www.syncoms.com* 
>
> *[image: LogoNEWmohLARGE]*
>
> *London*|*Bangalore*|*Orlando*
>
>
>
> *we innovate, plan, execute, and transform the business​*
>
>
>

Re: phoenix backup and recovery

[Jesse Yates]

You absolutely can use snapshots with phoenix.

You would need to snapshot both the phoenix metadata table and the table
you want to snapshot.

Then on restore, you restore both those tables to new tables, point phoenix
there and get the data you need.

Missing pieces:
1) I'm not sure there is a way to just copy over the rows you would need
from the metadata to bulk move the restored table back. This would be a
great contribution.
2) Schema changes need to be locked while doing the snapshot so you dont
see the wrong schema by accident. This may or may not exist - check the
docs. What you really want is a per-table schema lock, but a wholesale
schema lock would be sufficient, though slightly less clean.

On Fri, Nov 14, 2014, 8:59 AM Alex Kamil  wrote:

> Is it possible to use hbase snapshots
>  with phoenix?
> and what is the recommended process for backup and recovery
>
> thanks
> Alex
>

Re: anyone relying on hadoop1 still?

[Jesse Yates]

Yes. Hoya (which I think recently got renamed, but might be wrong there) is
running HBase inside various Yarn containers as a Yarn application. Idea
would be that yarn is the fabric for managing your entire cluster and you
just run various yarn apps, e.g. mapreduce, HBase, hive, etc
On Sep 25, 2014 10:59 PM, "Flavio Pompermaier"  wrote:

> So Hoya is a different thing from Hbase on hadoop 2?
> On Sep 25, 2014 11:23 PM, "Jesse Yates"  wrote:
>
>> Mostly its the use of hadoop metrics2 that has to change, as they change
>> the backing methods. If you are interested, take the hbase compatibility
>> layer <https://github.com/apache/hbase/tree/master/hbase-hadoop-compat/>
>> to see most of where HBase shims things.
>>
>> ---
>> Jesse Yates
>> @jesse_yates
>> jyates.github.com
>>
>> On Thu, Sep 25, 2014 at 2:17 PM, Flavio Pompermaier > > wrote:
>>
>>> Any help here? :(
>>> On Sep 24, 2014 8:29 AM, "Flavio Pompermaier" 
>>> wrote:
>>>
>>>> Just a curiosity..what is the difference between hbase on hadoop1 or
>>>> hadoop2 from a functional point of view?
>>>> Does HBase on hadoop2 (Hoya?) rely on YARN features?
>>>>
>>>> On Tue, Sep 23, 2014 at 8:15 PM, James Taylor 
>>>> wrote:
>>>>
>>>>> We'll definitely remove hadoop1 support from 4.x, as it's causing pom
>>>>> issues. We can't support both hadoop1 and hadoop2 and make hadoop2 the
>>>>> default (see prior discussions - basically due to how other projects
>>>>> poms are authored).
>>>>>
>>>>> Sounds like a few folks are still relying on hadoop1 for 3.x, so I
>>>>> guess we can leave support there a while longer.
>>>>>
>>>>> Thanks,
>>>>> James
>>>>>
>>>>> On Tue, Sep 23, 2014 at 10:54 AM, lars hofhansl 
>>>>> wrote:
>>>>> > At the very least we could make hadoop2 the default target in 4.x.
>>>>> > Seems fair to remove all the cruft from the 4 branch, too.
>>>>> >
>>>>> > There's still a fair amount of usage of HBase 0.94 on top of Hadoop
>>>>> 1.
>>>>> > So maybe keep it alive in 3.0? 3.0 can be retired when HBase 0.94 is
>>>>> retired (although I have no plans for 0.94 retirement, yet).
>>>>> >
>>>>> > -- Lars
>>>>> >
>>>>> >
>>>>> > - Original Message -
>>>>> > From: James Taylor 
>>>>> > To: "d...@phoenix.apache.org" ; user <
>>>>> user@phoenix.apache.org>
>>>>> > Cc:
>>>>> > Sent: Monday, September 22, 2014 10:19 PM
>>>>> > Subject: anyone relying on hadoop1 still?
>>>>> >
>>>>> > Hello,
>>>>> > We've been planning on dropping hadoop1 support for our 4.x releases
>>>>> > for a while now and it looks like it'll happen in 4.2. It'd be nice
>>>>> if
>>>>> > we could do the same for our 3.x releases, as the more similar the
>>>>> two
>>>>> > branches are, the less time it takes to keep them in sync.
>>>>> >
>>>>> > Is anyone out there still relying on hadoop1 support for future 3.x
>>>>> releases?
>>>>> >
>>>>> > Thanks,
>>>>> > James
>>>>> >
>>>>>
>>>>
>>>>
>>

Re: anyone relying on hadoop1 still?

[Jesse Yates]

Mostly its the use of hadoop metrics2 that has to change, as they change
the backing methods. If you are interested, take the hbase compatibility
layer <https://github.com/apache/hbase/tree/master/hbase-hadoop-compat/> to
see most of where HBase shims things.

---
Jesse Yates
@jesse_yates
jyates.github.com

On Thu, Sep 25, 2014 at 2:17 PM, Flavio Pompermaier 
wrote:

> Any help here? :(
> On Sep 24, 2014 8:29 AM, "Flavio Pompermaier" 
> wrote:
>
>> Just a curiosity..what is the difference between hbase on hadoop1 or
>> hadoop2 from a functional point of view?
>> Does HBase on hadoop2 (Hoya?) rely on YARN features?
>>
>> On Tue, Sep 23, 2014 at 8:15 PM, James Taylor 
>> wrote:
>>
>>> We'll definitely remove hadoop1 support from 4.x, as it's causing pom
>>> issues. We can't support both hadoop1 and hadoop2 and make hadoop2 the
>>> default (see prior discussions - basically due to how other projects
>>> poms are authored).
>>>
>>> Sounds like a few folks are still relying on hadoop1 for 3.x, so I
>>> guess we can leave support there a while longer.
>>>
>>> Thanks,
>>> James
>>>
>>> On Tue, Sep 23, 2014 at 10:54 AM, lars hofhansl 
>>> wrote:
>>> > At the very least we could make hadoop2 the default target in 4.x.
>>> > Seems fair to remove all the cruft from the 4 branch, too.
>>> >
>>> > There's still a fair amount of usage of HBase 0.94 on top of Hadoop 1.
>>> > So maybe keep it alive in 3.0? 3.0 can be retired when HBase 0.94 is
>>> retired (although I have no plans for 0.94 retirement, yet).
>>> >
>>> > -- Lars
>>> >
>>> >
>>> > - Original Message -
>>> > From: James Taylor 
>>> > To: "d...@phoenix.apache.org" ; user <
>>> user@phoenix.apache.org>
>>> > Cc:
>>> > Sent: Monday, September 22, 2014 10:19 PM
>>> > Subject: anyone relying on hadoop1 still?
>>> >
>>> > Hello,
>>> > We've been planning on dropping hadoop1 support for our 4.x releases
>>> > for a while now and it looks like it'll happen in 4.2. It'd be nice if
>>> > we could do the same for our 3.x releases, as the more similar the two
>>> > branches are, the less time it takes to keep them in sync.
>>> >
>>> > Is anyone out there still relying on hadoop1 support for future 3.x
>>> releases?
>>> >
>>> > Thanks,
>>> > James
>>> >
>>>
>>
>>

Re: Phoenix client maven dependencies

[Jesse Yates]

>From @Flavio

I think the jars on maven central are built against hadoop2, so it would be
> REALLY nice to publish both flavours in the next release


At the very least, the jars in maven should be what we build by default
(which, AFAIK, is hadoop1). HBase does some build magic so they can release
against both hadoop1 and hadoop2 by changing the version to include
-hadoopX in the name so they can different named artifacts for each when
they release. So obviously, its possible for us to do it too :)

The question to me is if we should even continue to support hadoop1 in
main? I don't think we should, though we could continue it in the 4.X line
for compatibility's sake.

---
Jesse Yates
@jesse_yates
jyates.github.com

On Mon, Sep 22, 2014 at 9:31 AM, James Taylor 
wrote:

> +1 to doing the same for hbase-testing-util. Thanks for the analysis,
> Andrew!
>
> James
>
> On Mon, Sep 22, 2014 at 9:18 AM, Andrew Purtell 
> wrote:
> > On Thu, Sep 18, 2014 at 3:01 PM, James Taylor 
> wrote:
> >> I see. That makes sense, but it's more of an HBase request than a
> >> Phoenix request. If HBase had a "client-only" pom, then Phoenix could
> >> have a "client-only" pom as well.
> >>
> >> Thanks,
> >> James
> >
> > If you specify only the 'hbase-client' Maven module as the HBase
> > dependency then this pulls in client-only deps (hbase-common,
> > hbase-protocol, commons-codec, commons-collections, commons-io,
> > commons-lang, commons-logging, guava, protobuf-java, netty, zookeeper,
> > htrace-core, jackson-mapper-asl, log4j)
> >
> > I think Phoenix might be pulling in all of the other HBase modules
> > because phoenix-core specifies the 'hbase-testing-util' HBase module
> > as a dependency, and not at test scope:
> >
> >   
> >   
> > org.apache.hbase
> > hbase-testing-util
> > 
> >   
> > org.jruby
> > jruby-complete
> >   
> > 
> >   
> > ...
> >
> > hbase-testing-util doesn't contain any code. It is a module you can
> > use that will pull in *everything* needed to start up mini cluster
> > tests, all of the HBase modules including hbase-server and the compat
> > modules, with compile scope. Maven doc says about compile scope: "This
> > is the default scope, used if none is specified. Compile dependencies
> > are available in all classpaths of a project. Furthermore, those
> > dependencies are propagated to dependent projects."
> >
> > Other test dependencies in the phoenix-core POM are included at test
> > scope and tagged as optional, e.g.
> >
> >   
> > org.apache.hadoop
> > hadoop-test
> > true
> > test
> >   
> >
> > Perhaps the same should be done for hbase-testing-util ?
> >
> >
> >> On Thu, Sep 18, 2014 at 1:52 PM, Flavio Pompermaier
> >>  wrote:
> >>> Because it is not clear which are the exact dependencies to add to a
> pom.xml
> >>> that uses the Phoenix client.
> >>> For example, why do I need to transitively hold all depency of
> hbase-server
> >>> in the client side or both hadoop (1 and 2) compatibility jars?!
> Wouldn't be
> >>> more easy to create a client jar that select only those needed classes
> of
> >>> such jar?
> >>>
> >>> On Sep 18, 2014 9:12 PM, "Mujtaba Chohan"  wrote:
> >>>>
> >>>> Falvio - Client jar is composed on multiple dependency jars extracted
> in
> >>>> one and is for convenience available in binary download only. This
> type of
> >>>> bundled jars are not supposed to go in maven repo. as maven
> automatically
> >>>> resolves required dependencies. To use maven repo., use the
> phoenix-core as
> >>>> artifact id and maven will pull in all the required dependencies for
> your
> >>>> project. Then if you want a single jar for your project, you can
> bundle your
> >>>> project jars in one.
> >>>>
> >>>> 
> >>>> org.apache.phoenix
> >>>> phoenix-core
> >>>> 4.1.0
> >>>> 
> >>>>
> >>>> //mujtaba
> >>>>
> >>>> On Thu, Sep 18, 2014 at 11:54 AM, Flavio Pompermaier
> >>>>  wrote

Re: Phoenix Meetups - Bay Area

[Jesse Yates]

No, but internally at Salesforce we actually just talked about doing one
recently.

I guess the question we need to ask is if people would actually come,
outside of the salesforce.com developers :)

if we can get even a handful of folks willing to commit, I'd say it would
be worth it!

---
Jesse Yates
@jesse_yates
jyates.github.com

On Mon, Sep 15, 2014 at 11:57 AM, Krishna  wrote:

> Hi, Is anyone aware of Phoenix meetups coming up in the next couple of
> months in Bay Area?
>
> Thanks
>
>
>
>

Re: Any possibility to drop SYSTEM.TRACING_STATS table?

[Jesse Yates]

You can specify your own table via the HBase configuration key
"phoenix.trace.statsTableName" everywhere the sink is used (client and
server). Then the stats table should switch over to using your table and
you can drop at will.

Currently, there is no way to drop the stats table through phoenix as its a
system table (though you could edit the metadata yourself in HBase and
remove those rows).

I just added PHOENIX-1240
<https://issues.apache.org/jira/browse/PHOENIX-1240> to help with not
having too much data in the tracing table by setting a TTL, which I think
is what you really have a concern with?

---
Jesse Yates
@jesse_yates
jyates.github.com


On Thu, Sep 4, 2014 at 1:34 AM, su...@certusnet.com.cn <
su...@certusnet.com.cn> wrote:

> I know that disable tracing would require to remove phoenix metrics2
> configuration
> and bouncing the region servers. But if I want to recreate the
> SYSTEM.TRACING_STATS table,
> what should I do? Noting that disable table from hbase shell would led
> phoniex table metadata
> into a bad state. Any suggesions?
> Thanks and Best Regards,
> Sun.
>
> --
> --
>
> CertusNet
>
>
>

Re: problem about using tracing

[Jesse Yates]

Filed https://issues.apache.org/jira/browse/PHOENIX-1234 and attached what
I think is the fix.

With this patch you still need to ensure that "hbase.zookeeper.quorum" gets
set to just the hostnames, not the host:port combination, as per the HBase
ref guide.

---
Jesse Yates
@jesse_yates
jyates.github.com


On Wed, Sep 3, 2014 at 1:31 PM, Jesse Yates  wrote:

> Actually, in hbase-site.xml, you would have
>
>   
>   hbase.zookeeper.property.clientPort
>   
> 
> 
>   hbase.zookeeper.quorum
>   
> rs1.example.com,rs2.example.com,rs3.example.com,rs4.example.com,rs5.example.com
> 
>
> from the ref guide, zookeeper
> <http://hbase.apache.org/book.html#zookeeper> section
>
> which then gets resolved to something like: "rs1.example.com:,
> rs2.example.com:,rs3.example.com:,rs4.example.com:,
> rs5.example.com:" from ZKConfig. Then when phoenix tries to parse it
> in QueryUtil, it gets:
>
>> server=rs1.example.com:,rs2.example.com:,rs3.example.com:,
>> rs4.example.com:,rs5.example.com:
>
>  port=
>
>
> Which will not create a correct connection.
>
>
> ---
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Wed, Sep 3, 2014 at 1:22 PM, Jeffrey Zhong 
> wrote:
>
>>
>> I think the “hbase.zookeeper.quorum” in the hbase-site.xml with value
>> like "zookeeper2:2181,zookeeper1:2181,zookeeper3:2181” while
>> Phoenix doesn’t support this format. We had this discussion on the topic at
>> https://issues.apache.org/jira/browse/PHOENIX-883.
>>
>> To resolve the following issue, user can just change the 
>> hbase.zookeeper.quorum
>> to zookeeper2,zookeeper1,zookeeper3
>>
>> From: Mujtaba Chohan 
>> Reply-To: 
>> Date: Wednesday, September 3, 2014 at 1:05 PM
>> To: 
>> Subject: Re: problem about using tracing
>>
>> Phoenix connection URL should be of this form
>> jdbc:phoenix:zookeeper2,zookeeper1,zookeeper3:2181
>>
>>
>>
>> On Wed, Sep 3, 2014 at 12:11 PM, Jesse Yates 
>> wrote:
>>
>>> It looks like the connection string that the tracing module is using
>>> isn't configured correctly. Is 2181 the client port on which you are
>>> running zookeeper?
>>>
>>> @James Taylor - phoenix can connect to multiple ZK nodes this way, right?
>>>
>>> ---
>>> Jesse Yates
>>> @jesse_yates
>>> jyates.github.com
>>>
>>>
>>> On Wed, Sep 3, 2014 at 12:59 AM, su...@certusnet.com.cn <
>>> su...@certusnet.com.cn> wrote:
>>>
>>>> Hi all,
>>>> I am trying to facilitate tracing according to the instructions here
>>>> <http://phoenix.apache.org/tracing.html>. Here are my several
>>>> operations:
>>>> 1. copy the phoenix-hadoop2-compat/bin/ attributes files into my hbase
>>>> classpath($HBASE_HOME/conf)
>>>> 2. modify hbase-site.xml and adding the following properties:
>>>> 
>>>>  phoenix.trace.frequency
>>>>  always
>>>>  
>>>> 3. restart hbase cluster and run phoenix through sqlline client:
>>>>   ./bin/sqlline.py zookeeper1,zookeeper2,zookeeper3
>>>>as zookeeper1,zookeeper2,zookeeper3 are my zookeeper hosts
>>>> 4. When I am trying to see the tracing feature thourgh sqlline query as
>>>> the following:
>>>>   select count (*) from mytable;
>>>> I checked the regionserver log and found the following exception.
>>>> Any available hints?
>>>>
>>>>2014-09-03 15:40:53,218 ERROR [tracing] impl.MetricsSinkAdapter:
>>>> Got sink exception and over retry limit, suppressing further error messages
>>>> java.lang.RuntimeException: java.sql.SQLException: ERROR 102 (08001):
>>>> Malformed connection url.
>>>> jdbc:phoenix:zookeeper2:2181,zookeeper1:2181,zookeeper3:2181;
>>>> at
>>>> org.apache.phoenix.trace.PhoenixTableMetricsWriter.lazyInitialize(PhoenixTableMetricsWriter.java:110)
>>>>
>>>> at
>>>> org.apache.phoenix.trace.PhoenixTableMetricsWriter.addMetrics(PhoenixTableMetricsWriter.java:185)
>>>>
>>>> at
>>>> org.apache.phoenix.trace.PhoenixMetricsSink.putMetrics(PhoenixMetricsSink.java:92)
>>>>
>>>> at
>>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:173)
>>>>
>>>> at
>&

Re: problem about using tracing

[Jesse Yates]

Actually, in hbase-site.xml, you would have

  
  hbase.zookeeper.property.clientPort
  


  hbase.zookeeper.quorum
  
rs1.example.com,rs2.example.com,rs3.example.com,rs4.example.com,rs5.example.com


from the ref guide, zookeeper <http://hbase.apache.org/book.html#zookeeper>
section

which then gets resolved to something like: "rs1.example.com:,
rs2.example.com:,rs3.example.com:,rs4.example.com:,
rs5.example.com:" from ZKConfig. Then when phoenix tries to parse it in
QueryUtil, it gets:

> server=rs1.example.com:,rs2.example.com:,rs3.example.com:,
> rs4.example.com:,rs5.example.com:

 port=


Which will not create a correct connection.


---
Jesse Yates
@jesse_yates
jyates.github.com


On Wed, Sep 3, 2014 at 1:22 PM, Jeffrey Zhong 
wrote:

>
> I think the “hbase.zookeeper.quorum” in the hbase-site.xml with value
> like "zookeeper2:2181,zookeeper1:2181,zookeeper3:2181” while
> Phoenix doesn’t support this format. We had this discussion on the topic at
> https://issues.apache.org/jira/browse/PHOENIX-883.
>
> To resolve the following issue, user can just change the 
> hbase.zookeeper.quorum
> to zookeeper2,zookeeper1,zookeeper3
>
> From: Mujtaba Chohan 
> Reply-To: 
> Date: Wednesday, September 3, 2014 at 1:05 PM
> To: 
> Subject: Re: problem about using tracing
>
> Phoenix connection URL should be of this form
> jdbc:phoenix:zookeeper2,zookeeper1,zookeeper3:2181
>
>
>
> On Wed, Sep 3, 2014 at 12:11 PM, Jesse Yates 
> wrote:
>
>> It looks like the connection string that the tracing module is using
>> isn't configured correctly. Is 2181 the client port on which you are
>> running zookeeper?
>>
>> @James Taylor - phoenix can connect to multiple ZK nodes this way, right?
>>
>> ---
>> Jesse Yates
>> @jesse_yates
>> jyates.github.com
>>
>>
>> On Wed, Sep 3, 2014 at 12:59 AM, su...@certusnet.com.cn <
>> su...@certusnet.com.cn> wrote:
>>
>>> Hi all,
>>> I am trying to facilitate tracing according to the instructions here
>>> <http://phoenix.apache.org/tracing.html>. Here are my several
>>> operations:
>>> 1. copy the phoenix-hadoop2-compat/bin/ attributes files into my hbase
>>> classpath($HBASE_HOME/conf)
>>> 2. modify hbase-site.xml and adding the following properties:
>>> 
>>>  phoenix.trace.frequency
>>>  always
>>>  
>>> 3. restart hbase cluster and run phoenix through sqlline client:
>>>   ./bin/sqlline.py zookeeper1,zookeeper2,zookeeper3
>>>as zookeeper1,zookeeper2,zookeeper3 are my zookeeper hosts
>>> 4. When I am trying to see the tracing feature thourgh sqlline query as
>>> the following:
>>>   select count (*) from mytable;
>>> I checked the regionserver log and found the following exception.
>>> Any available hints?
>>>
>>>2014-09-03 15:40:53,218 ERROR [tracing] impl.MetricsSinkAdapter: Got
>>> sink exception and over retry limit, suppressing further error messages
>>> java.lang.RuntimeException: java.sql.SQLException: ERROR 102 (08001):
>>> Malformed connection url.
>>> jdbc:phoenix:zookeeper2:2181,zookeeper1:2181,zookeeper3:2181;
>>> at
>>> org.apache.phoenix.trace.PhoenixTableMetricsWriter.lazyInitialize(PhoenixTableMetricsWriter.java:110)
>>>
>>> at
>>> org.apache.phoenix.trace.PhoenixTableMetricsWriter.addMetrics(PhoenixTableMetricsWriter.java:185)
>>>
>>> at
>>> org.apache.phoenix.trace.PhoenixMetricsSink.putMetrics(PhoenixMetricsSink.java:92)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:173)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:41)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:127)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:86)
>>>
>>> Caused by: java.sql.SQLException: ERROR 102 (08001): Malformed
>>> connection url.
>>> jdbc:phoenix:zookeeper2:2181,zookeeper1:2181,zookeeper3:2181;
>>> at
>>> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:333)
>>>
>>> at
>>> org.apache.phoenix.exception.SQLExceptionI

Re: problem about using tracing

[Jesse Yates]

Looks like that actually might be a bug. HBase's ZKConfig returns a
host:port, comma separated string which we then put as the connection URL.
Never caught in mini-clusters since we only ever return a single ZK.

I'll see if I can reproduce and then let you all know when/if I file JIRA

---
Jesse Yates
@jesse_yates
jyates.github.com


On Wed, Sep 3, 2014 at 1:05 PM, Mujtaba Chohan  wrote:

> Phoenix connection URL should be of this form
> jdbc:phoenix:zookeeper2,zookeeper1,zookeeper3:2181
>
>
>
> On Wed, Sep 3, 2014 at 12:11 PM, Jesse Yates 
> wrote:
>
>> It looks like the connection string that the tracing module is using
>> isn't configured correctly. Is 2181 the client port on which you are
>> running zookeeper?
>>
>> @James Taylor - phoenix can connect to multiple ZK nodes this way, right?
>>
>> ---
>> Jesse Yates
>> @jesse_yates
>> jyates.github.com
>>
>>
>> On Wed, Sep 3, 2014 at 12:59 AM, su...@certusnet.com.cn <
>> su...@certusnet.com.cn> wrote:
>>
>>> Hi all,
>>> I am trying to facilitate tracing according to the instructions here
>>> <http://phoenix.apache.org/tracing.html>. Here are my several
>>> operations:
>>> 1. copy the phoenix-hadoop2-compat/bin/ attributes files into my hbase
>>> classpath($HBASE_HOME/conf)
>>> 2. modify hbase-site.xml and adding the following properties:
>>> 
>>>  phoenix.trace.frequency
>>>  always
>>>  
>>> 3. restart hbase cluster and run phoenix through sqlline client:
>>>   ./bin/sqlline.py zookeeper1,zookeeper2,zookeeper3
>>>as zookeeper1,zookeeper2,zookeeper3 are my zookeeper hosts
>>> 4. When I am trying to see the tracing feature thourgh sqlline query as
>>> the following:
>>>   select count (*) from mytable;
>>> I checked the regionserver log and found the following exception.
>>> Any available hints?
>>>
>>>2014-09-03 15:40:53,218 ERROR [tracing] impl.MetricsSinkAdapter: Got
>>> sink exception and over retry limit, suppressing further error messages
>>> java.lang.RuntimeException: java.sql.SQLException: ERROR 102 (08001):
>>> Malformed connection url.
>>> jdbc:phoenix:zookeeper2:2181,zookeeper1:2181,zookeeper3:2181;
>>> at
>>> org.apache.phoenix.trace.PhoenixTableMetricsWriter.lazyInitialize(PhoenixTableMetricsWriter.java:110)
>>>
>>> at
>>> org.apache.phoenix.trace.PhoenixTableMetricsWriter.addMetrics(PhoenixTableMetricsWriter.java:185)
>>>
>>> at
>>> org.apache.phoenix.trace.PhoenixMetricsSink.putMetrics(PhoenixMetricsSink.java:92)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:173)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:41)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:127)
>>>
>>> at
>>> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:86)
>>>
>>> Caused by: java.sql.SQLException: ERROR 102 (08001): Malformed
>>> connection url.
>>> jdbc:phoenix:zookeeper2:2181,zookeeper1:2181,zookeeper3:2181;
>>> at
>>> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:333)
>>>
>>> at
>>> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:133)
>>>
>>> at
>>> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.getMalFormedUrlException(PhoenixEmbeddedDriver.java:183)
>>>
>>> at
>>> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.create(PhoenixEmbeddedDriver.java:238)
>>>
>>> at
>>> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:144)
>>>
>>> at
>>> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:129)
>>>
>>> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:187)
>>> at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:213)
>>> at
>>> org.apache.phoenix.trace.PhoenixTableMetricsWriter.lazyInitialize(PhoenixTableMetricsWriter.java:100)
>>>
>>> ... 7 more
>>>
>>> --
>>> --
>>>
>>> CertusNet
>>>
>>>
>>>
>>
>

Re: problem about using tracing

[Jesse Yates]

It looks like the connection string that the tracing module is using isn't
configured correctly. Is 2181 the client port on which you are running
zookeeper?

@James Taylor - phoenix can connect to multiple ZK nodes this way, right?

---
Jesse Yates
@jesse_yates
jyates.github.com


On Wed, Sep 3, 2014 at 12:59 AM, su...@certusnet.com.cn <
su...@certusnet.com.cn> wrote:

> Hi all,
> I am trying to facilitate tracing according to the instructions here
> <http://phoenix.apache.org/tracing.html>. Here are my several operations:
> 1. copy the phoenix-hadoop2-compat/bin/ attributes files into my hbase
> classpath($HBASE_HOME/conf)
> 2. modify hbase-site.xml and adding the following properties:
> 
>  phoenix.trace.frequency
>  always
>  
> 3. restart hbase cluster and run phoenix through sqlline client:
>   ./bin/sqlline.py zookeeper1,zookeeper2,zookeeper3
>  as zookeeper1,zookeeper2,zookeeper3 are my zookeeper hosts
> 4. When I am trying to see the tracing feature thourgh sqlline query as
> the following:
>   select count (*) from mytable;
> I checked the regionserver log and found the following exception. Any
> available hints?
>
>2014-09-03 15:40:53,218 ERROR [tracing] impl.MetricsSinkAdapter: Got
> sink exception and over retry limit, suppressing further error messages
> java.lang.RuntimeException: java.sql.SQLException: ERROR 102 (08001):
> Malformed connection url.
> jdbc:phoenix:zookeeper2:2181,zookeeper1:2181,zookeeper3:2181;
> at
> org.apache.phoenix.trace.PhoenixTableMetricsWriter.lazyInitialize(PhoenixTableMetricsWriter.java:110)
>
> at
> org.apache.phoenix.trace.PhoenixTableMetricsWriter.addMetrics(PhoenixTableMetricsWriter.java:185)
>
> at
> org.apache.phoenix.trace.PhoenixMetricsSink.putMetrics(PhoenixMetricsSink.java:92)
>
> at
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:173)
>
> at
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:41)
>
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:127)
>
> at
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:86)
>
> Caused by: java.sql.SQLException: ERROR 102 (08001): Malformed connection
> url. jdbc:phoenix:zookeeper2:2181,zookeeper1:2181,zookeeper3:2181;
> at
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:333)
>
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:133)
>
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.getMalFormedUrlException(PhoenixEmbeddedDriver.java:183)
>
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.create(PhoenixEmbeddedDriver.java:238)
>
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:144)
>
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:129)
>
> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)
> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> at java.sql.DriverManager.getConnection(DriverManager.java:187)
> at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:213)
> at
> org.apache.phoenix.trace.PhoenixTableMetricsWriter.lazyInitialize(PhoenixTableMetricsWriter.java:100)
>
> ... 7 more
>
> --
> --
>
> CertusNet
>
>
>

Re: Tracing Q's

[Jesse Yates]

To start with, there are a bunch of things we are planning with tracing:
https://issues.apache.org/jira/browse/PHOENIX-1121

But to answer your questions,


> Can we use something like zipkin-htrace adapter for Phoenix traces? And if
> I did would the calls be coming from the RS?


Yes, we could roll in something like that as well, but there just isn't a
knob for it right now. You would need the same config on the server (all
the tracing config goes through the same interface though, so it should be
too hard). Right now, its all being written to an HBase table from both
sides of the request, so you could pull that in later to populate zipkin as
all.

We could also add a span receiver to write to zipkin. I'd be more inclinded
to write to zipkin from the phoenix table as thats more likely to be stable
storage. All the same information would be there, but I'd trust my HBase
tables :)

* How do you get the trace id on a query you create?


If there is something you are looking to trace, you could actually create a
trace before creating your phoenix request, and pull the traceID out of
there (you could also add any annotations you wanted, like the app server's
request id) Phoenix will either continue the trace, if one is started, or
start a new one, if configured to do so.

Starting a new one is generally just for introspection into a running
system to see how things are doing. It wouldn't be tied to anything in
particular. There is some pending work in the above mentioned JIRA for
adding tag (timeline annotations, in HTrace parlance)/annotations
(key-value annotations) to a phoenix request/connection, but you should be
able to do what you want just by starting the trace before making the
phoenix request. If phoenix is configured correctly, it should just work
with the rest of the phoenix trace sink infrastructure

 Do you have to load the DDL manually


Nope, its part of the PhoenixTableMetricsWriter, here
<https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/trace/PhoenixTableMetricsWriter.java#L142>.
When it receives a metric (really, just a conversion of a span to a Hadoop
metrics2 metric), it will create the table as needed.

Hope that helps!

---
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Aug 26, 2014 at 7:21 PM, Dan Di Spaltro 
wrote:

> I've used the concept of tracing quite a bit in previous projects and I
> had a couple questions:
>
> * Can we use something like zipkin-htrace adapter for Phoenix traces? And
> if I did would the calls be coming from the RS?
> * How do you get the trace id on a query you create?  Generally I've used
> something where I can log back to a client a trace/span, and then go look
> through the queries to match up why something took so long, etc. I could be
> thinking about this wrong...
> * Do you have to load the DDL manually, nothing seems to auto-create it,
> no system table seems to be created outside of sequences and tables.  I
> have the default config files from Phoenix on the classpath.  I also have
> the compat and server jars on the CP.  Below are the log lines I see in the
> master and regionserver.
>   - I have set props.setProperty("phoenix.trace.frequency", "always") for
> every query.
>
> 2014-08-27 01:55:27,483 INFO  [main] trace.PhoenixMetricsSink: Writing
> tracing metrics to phoenix table
> 2014-08-27 01:55:27,484 INFO  [main] trace.PhoenixMetricsSink:
> Instantiating writer class:
> org.apache.phoenix.trace.PhoenixTableMetricsWriter
> 2014-08-27 01:55:27,490 INFO  [main] trace.PhoenixTableMetricsWriter:
> Phoenix tracing writer started
>
> Thanks for the help,
>
> -Dan
>
> --
> Dan Di Spaltro
>

Re: tracing

[Jesse Yates]

You will need a build of phoenix that uses the -Dhadoop.profile=2 property.
If that isn't around, we should add it to the releases to differentiate it
from the standard, hadoop1 support jars (seems like not).

Take a look at the source (specifically in the phoenix-hadoop2-compat/bin
folder) and you can pull the files you need directly from there for the
moment.

---
Jesse Yates
@jesse_yates
jyates.github.com


On Fri, Aug 22, 2014 at 2:35 AM, su...@certusnet.com.cn <
su...@certusnet.com.cn> wrote:

> Hi all,
> I got the v4.1.0-rc0 phoenix release from here:
> https://github.com/apache/phoenix/releases
> while trying to facilitate the tracing (
> http://phoenix.apache.org/tracing.html ).
> However, I got really confused from the instructions. For example, the
> configuration here
> as the following chart illustrated two files in the bin/ directory. But I
> did not catch any
> *such files in the $PHOENIX_HOME/bin directory if I was correctly
> understanding the right*
> *bin/ directory. *
>
> Hoping anyone can provide available experience about the tracing usage.
>
> Thanks and Best Regards,
> Sun
>
>
>

Re: Error when using Pig Storage: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

[Jesse Yates]

FWIW internally at Salesforce we also patch the HBase and Hadoop poms to
support our own internal 'light forks'. Its really not a big deal to manage
- a couple of jenkins jobs (one to automate, one to track open source
changes and ensure your patch(es) still work, etc) and you are good to go.

I imagine this is also what various distributors are doing for their forks
as well.

---
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney 
wrote:

> First of all, I apologize if you feel like I was picking on you. I was not
> trying to do that.
>
> My understanding is that Salesforce pays people to work on Phoenix. Is
> that not the case? I'm hoping one of them will add spark-like support for
> CDH and HDP to advance the project.
>
> And I don't mention the POM thing to pick on someone. The majority of
> HBase users are not going to be able to use Phoenix because they run a
> commercial distribution of Hadoop and aren't pom wizards. That seems kind
> of important for the well being of the project.
> ᐧ
>
>
> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell 
> wrote:
>
>> I don't think an Apache project should spend precious bandwidth tracking
>> the various and sundry redistributors of Apache ecosystem projects. This is
>> putting the cart before the horse. The horse is the Apache upstream
>> projects. The cart is the commercial distributions leveraging the Apache
>> ecosystem for profit. Spark is not a good example, it is supported by a
>> commercial concern, Databricks. What commercial company supports Phoenix?
>> Why do you think it is appropriate to pick on volunteers because editing
>> POM files is too much work?
>>
>>
>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney > > wrote:
>>
>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185
>>> because requiring users to hand-edit the pom.xml just to build against CDH
>>> and HDP is nuts.
>>> ᐧ
>>>
>>>
>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney <
>>> russell.jur...@gmail.com> wrote:
>>>
>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 and
>>>> https://issues.apache.org/jira/browse/PHOENIX-1184
>>>>
>>>> I hope this can be resolved soon, we are pretty screwed right now.
>>>> ᐧ
>>>>
>>>>
>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates 
>>>> wrote:
>>>>
>>>>> Yup, that looks like an issue to me :-/
>>>>>
>>>>> ---
>>>>> Jesse Yates
>>>>> @jesse_yates
>>>>> jyates.github.com
>>>>>
>>>>>
>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney <
>>>>> russell.jur...@gmail.com> wrote:
>>>>>
>>>>>> Running against any version would be ok, but it does not work. I get
>>>>>> this error:
>>>>>>
>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error 
>>>>>> running child : java.lang.IncompatibleClassChangeError: Found interface 
>>>>>> org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
>>>>>>
>>>>>> ᐧ
>>>>>> I went back and checked out the Phoenix commit before the pom.xml
>>>>>> was upgraded to 0.98.4
>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>,
>>>>>> and I get the same error.
>>>>>>
>>>>>> Not sure what to do. Stuck. I imagine most users must be too, as CDH
>>>>>> 5 is probably the most popular installation there is?
>>>>>>
>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor <
>>>>>> jamestay...@apache.org> wrote:
>>>>>>
>>>>>>>  The dependencies on HBase 0.98.4 are *compile time* dependencies.
>>>>>>> Is it necessary for you to compile against CDH 5.1 or just run against 
>>>>>>> it?
>>>>>>>
>>>>>>>
>>>>>>> On Tuesday, August 19, 2014, Russell Jurney <
>>>>>>> russell.jur...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix? How can
>>>>>>>> this be fixed? I'm not sure what to do. We're in limbo on our new 
>>>

Re: Error when using Pig Storage: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

[Jesse Yates]

Yup, that looks like an issue to me :-/

---
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney 
wrote:

> Running against any version would be ok, but it does not work. I get this
> error:
>
> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.IncompatibleClassChangeError: Found interface 
> org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
>
> ᐧ
> I went back and checked out the Phoenix commit before the pom.xml was
> upgraded to 0.98.4
> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>,
> and I get the same error.
>
> Not sure what to do. Stuck. I imagine most users must be too, as CDH 5 is
> probably the most popular installation there is?
>
> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor 
> wrote:
>
>>  The dependencies on HBase 0.98.4 are *compile time* dependencies. Is it
>> necessary for you to compile against CDH 5.1 or just run against it?
>>
>>
>> On Tuesday, August 19, 2014, Russell Jurney 
>> wrote:
>>
>>> Thats really bad. That means... CDH 5.x can't run Phoenix? How can this
>>> be fixed? I'm not sure what to do. We're in limbo on our new cluster now.
>>>
>>>
>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran 
>>> wrote:
>>>
>>>> Hi Russel,
>>>> Apparently, Phoenix 4.0.0 leverages few API methods of HBase 0.98.4
>>>> v  which aren't present within 0.98.1 that comes with CDH 5.1 . That's the
>>>> primary cause for the build issues.
>>>>
>>>> Regards
>>>> Ravi
>>>>
>>>>
>>>>
>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney <
>>>> russell.jur...@gmail.com> wrote:
>>>>
>>>>> Talking to myself, but hopefully creating good docs. Replacing the
>>>>> previous hadoop version with one I found here:
>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/,
>>>>>  2.3.0-mr1-cdh5.1.0,
>>>>> makes things get a little further.
>>>>>
>>>>> I can't get past some build errors, however. Has anyone done this
>>>>> before me who can help?
>>>>>
>>>>> [ERROR]
>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>> cannot find symbol
>>>>>
>>>>>   symbol:   method valueOf(java.lang.String,int,int)
>>>>>
>>>>>   location: class org.apache.hadoop.hbase.ServerName
>>>>>
>>>>>
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile
>>>>> (default-compile) on project phoenix-core: Compilation failure
>>>>>
>>>>> [ERROR]
>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>> cannot find symbol
>>>>>
>>>>> [ERROR] symbol:   method valueOf(java.lang.String,int,int)
>>>>>
>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName
>>>>>
>>>>> [ERROR] -> [Help 1]
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney <
>>>>> russell.jur...@gmail.com> wrote:
>>>>>
>>>>>> Ok, so it is clear to me what I have to do. I have to edit my pom.xml
>>>>>> to point at CDH 5.1, which translates into:
>>>>>>
>>>>>> Add the cloudera repo:
>>>>>>
>>>>>> 
>>>>>>   cloudera
>>>>>>   
>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/
>>>>>> 
>>>>>>
>>>>>>
>>>>>> Then change the hadoop and hbase versions:
>>>>>>
>>>>>>  
>>>>>> 0.98.1-cdh5.1.0
>>>>>> 0.98.1-cdh5.1.0
>>>>>> 2.3.0-cdh5.1.0
>>>>>> 2.3.0-cdh5.1.0
>>>>>>
>>>>>>
>>>>>> However, I get this error when I build, which tells me there is m

Re: Error when using Pig Storage: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

[Jesse Yates]

@James Taylor, correct me if I'm wrong, but it should be backwards
compatible with older versions of HBase 0.98 - the build issues are
separate from phoenix actually being able to run on that cluster
(compatibilities should be handled via reflection).


If there are cases where that's not true, means you should probably file a
jira.

---
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Aug 19, 2014 at 11:36 AM, Russell Jurney 
wrote:

> Thats really bad. That means... CDH 5.x can't run Phoenix? How can this be
> fixed? I'm not sure what to do. We're in limbo on our new cluster now.
> ᐧ
>
>
> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran 
> wrote:
>
>> Hi Russel,
>> Apparently, Phoenix 4.0.0 leverages few API methods of HBase 0.98.4
>> v  which aren't present within 0.98.1 that comes with CDH 5.1 . That's the
>> primary cause for the build issues.
>>
>> Regards
>> Ravi
>>
>>
>>
>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney > > wrote:
>>
>>> Talking to myself, but hopefully creating good docs. Replacing the
>>> previous hadoop version with one I found here:
>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/,
>>>  2.3.0-mr1-cdh5.1.0,
>>> makes things get a little further.
>>>
>>> I can't get past some build errors, however. Has anyone done this before
>>> me who can help?
>>>
>>> [ERROR]
>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>> cannot find symbol
>>>
>>>   symbol:   method valueOf(java.lang.String,int,int)
>>>
>>>   location: class org.apache.hadoop.hbase.ServerName
>>>
>>>
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile
>>> (default-compile) on project phoenix-core: Compilation failure
>>>
>>> [ERROR]
>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>> cannot find symbol
>>>
>>> [ERROR] symbol:   method valueOf(java.lang.String,int,int)
>>>
>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName
>>>
>>> [ERROR] -> [Help 1]
>>>
>>> ᐧ
>>>
>>>
>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney <
>>> russell.jur...@gmail.com> wrote:
>>>
>>>> Ok, so it is clear to me what I have to do. I have to edit my pom.xml
>>>> to point at CDH 5.1, which translates into:
>>>>
>>>> Add the cloudera repo:
>>>>
>>>> 
>>>>   cloudera
>>>>   https://repository.cloudera.com/artifactory/cloudera-repos/
>>>> 
>>>> 
>>>>
>>>>
>>>> Then change the hadoop and hbase versions:
>>>>
>>>>  
>>>> 0.98.1-cdh5.1.0
>>>> 0.98.1-cdh5.1.0
>>>> 2.3.0-cdh5.1.0
>>>> 2.3.0-cdh5.1.0
>>>>
>>>>
>>>> However, I get this error when I build, which tells me there is more
>>>> complex POM surgery required.
>>>>
>>>> [ERROR] Failed to execute goal on project phoenix-core: Could not
>>>> resolve dependencies for project
>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating: The following
>>>> artifacts could not be resolved:
>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0,
>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0: Could not find artifact
>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0 in apache release (
>>>> https://repository.apache.org/content/repositories/releases/) -> [Help
>>>> 1]
>>>>
>>>> Beyond changing the versions, I do not know how to fix this. Can anyone
>>>> help?
>>>>
>>>> In general, is it possible to actually handle different CDH versions in
>>>> this project? One shouldn't have to do pom surgery to build Phoenix for the
>>>> most common platform.
>>>> ᐧ
>>>>
>>>>
>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney <
>>>> russell.jur...@gmail.com> wrote:
>>>>
>>>>> When I t

Re: Error when try to create mutable secondary index...

[Jesse Yates]

That seems correct. I'm not sure where the issue is either. It seems like
the property isn't in the correct config files (also, you don't need it on
the master configs, but it won't hurt).

Is the property there when you dump the config from the RS's UI page?

---
Jesse Yates
@jesse_yates
jyates.github.com


On Mon, Aug 11, 2014 at 10:27 AM, Saravanan A 
wrote:

> No am not sure where the issue is...
>
> Procedure i did for installation for Phoenix installation:
>
> 1.Extracted Phoenix-3.0.
> 2.Added Phoenix core.jar in all region servers and in master..
> 3. Added this 
>
>
>   hbase.regionserver.wal.codec
>   org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
>
>  property in all hbase-site.xml file in region servers,master
> and in phoenix bin dir..
> 4.Restarted hbase.
>
> is this right or am missing anything???
>
>
>
>
> On Mon, Aug 11, 2014 at 10:38 PM, Jesse Yates 
> wrote:
>
>> Well now, that is strange. Maybe its something to do with CDH? Have you
>> talked to those fellas? Or maybe someone from Cloudera has an insight?
>>
>> Seems like it should work
>> On Aug 11, 2014 9:55 AM, "Saravanan A"  wrote:
>>
>>> *bin/hbase classpath:*
>>>
>>>
>>> */opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../conf:/usr/java/default/lib/tools.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/..:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../hbase-0.94.15-cdh4.7.0-security.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../hbase-0.94.15-cdh4.7.0-security-tests.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../hbase.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/asm-3.2.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/avro-1.7.4.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/cloudera-jets3t-2.0.0-cdh4.7.0.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-collections-3.2.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-configuration-1.6.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-daemon-1.0.3.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-digester-1.8.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-io-2.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-lang-2.5.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/core-3.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/grizzly-framework-2.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/grizzly-framework-2.1.1-tests.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/grizzly-http-2.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/grizzly-http-server-2.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/grizzly-http-servlet-2.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/grizzly-rcm-2.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/guice-3.0.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/guice-servlet-3.0.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/hamcrest-core-1.3.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hbase/bin/../lib/*
>>> *high-scale-lib-1.1.1.jar:/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/h

Re: Error when try to create mutable secondary index...

[Jesse Yates]

; *
>
> this is the result i got for Hbase classpath command...and this is the
> "/opt/cloudera/parcels/CDH/lib/hbase/lib/" path i executed the code...
>
>
> On Mon, Aug 11, 2014 at 9:29 PM, Jesse Yates 
> wrote:
>
>> The classpath you are using above doesn't include the HBase config files,
>> so the code executed will correctly tell you that the class exists, but is
>> not configured.
>>
>> Have you tried running
>> "bin/hbase classpath"
>> to see what you're classpath is at RS startup? If its the same as the -cp
>> argument, its missing the config files.
>> On Aug 11, 2014 6:10 AM, "Saravanan A"  wrote:
>>
>>> *This is the command i run in hbase classpath (test1.jar is my jar file)*:
>>> hbase -cp
>>> .:hadoop-common-2.0.0-cdh4.7.0.jar:commons-logging-1.1.1.jar:hbase-0.94.15-cdh4.7.0-security.jar:com.google.collections.jar:commons-collections-3.2.1.jar:phoenix-core-3.0.0-incubating.jar:com.google.guava_1.6.0.jar:test1.jar
>>> FixConfigFile
>>>
>>> *The Output:*
>>> Found
>>> Not Found
>>>
>>> *This is my full code:*
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>>
>>> public class FixConfigFile {
>>>
>>> public static final String INDEX_WAL_EDIT_CODEC_CLASS_NAME =
>>> "org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec";
>>>  public static final String WAL_EDIT_CODEC_CLASS_KEY =
>>> "org.apache.hadoop.hbase.regionserver.wal.codec";
>>>  public static void main(String[] args) {
>>> Configuration config=new Configuration();
>>> isWALEditCodecSet(config);
>>>
>>> }
>>> public static boolean isWALEditCodecSet(Configuration conf) {
>>> // check to see if the WALEditCodec is installed
>>> try {
>>> // Use reflection to load the IndexedWALEditCodec, since it
>>> may not load with an older version
>>> // of HBase
>>> Class.forName(INDEX_WAL_EDIT_CODEC_CLASS_NAME);
>>> System.out.println("Found");
>>> } catch (Throwable t) {
>>> System.out.println("Error");
>>> return false;
>>> }
>>> if
>>> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
>>> null))) {
>>> // its installed, and it can handle compression and
>>> non-compression cases
>>> System.out.println("True");
>>> return true;
>>> }
>>> System.out.println("Not Found");
>>> return false;
>>> }
>>>
>>> }
>>> 
>>>
>>> am not sure this is how you want me to execute the code...If am wrong
>>> please guide me...
>>>
>>>
>>>
>>> On Sat, Aug 9, 2014 at 8:32 PM, Jesse Yates 
>>> wrote:
>>>
>>>> When you run
>>>>$ bin/hbase classpath
>>>> What do you get? Should help illuminate if everything is setup right.
>>>>
>>>> If the phoenix jar is there, then check the contents of the jar (
>>>> http://docs.oracle.com/javase/tutorial/deployment/jar/view.html) and
>>>> make sure the classes are present.
>>>>  On Aug 9, 2014 1:03 AM, "Saravanan A" 
>>>> wrote:
>>>>
>>>>> Hi Jesse,
>>>>>
>>>>> I ran the following code to test the existence of the classes you
>>>>> asked me to check. I initialized the two constants to the following 
>>>>> values.
>>>>>
>>>>> ===
>>>>> public static final String INDEX_WAL_EDIT_CODEC_CLASS_NAME =
>>>>> "org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec";
>>>>>
>>>>> public static final String WAL_EDIT_CODEC_CLASS_KEY =
>>>>> "hbase.regionserver.wal.codec";
>>>>> ==
>>>>>
>>>>> Then I ran the following code and got the error "Not found" in the
>>>>> equality test.
>>>>>
>>>>> 
>>>>> if
>>>>> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
>>>>> null))) {
>>>>> // its installed, and it can handle compression and
>>>>> non-compression cases
>>

Re: Error when try to create mutable secondary index...

[Jesse Yates]

The classpath you are using above doesn't include the HBase config files,
so the code executed will correctly tell you that the class exists, but is
not configured.

Have you tried running
"bin/hbase classpath"
to see what you're classpath is at RS startup? If its the same as the -cp
argument, its missing the config files.
On Aug 11, 2014 6:10 AM, "Saravanan A"  wrote:

> *This is the command i run in hbase classpath (test1.jar is my jar file)*:
> hbase -cp
> .:hadoop-common-2.0.0-cdh4.7.0.jar:commons-logging-1.1.1.jar:hbase-0.94.15-cdh4.7.0-security.jar:com.google.collections.jar:commons-collections-3.2.1.jar:phoenix-core-3.0.0-incubating.jar:com.google.guava_1.6.0.jar:test1.jar
> FixConfigFile
>
> *The Output:*
> Found
> Not Found
>
> *This is my full code:*
>
> import org.apache.hadoop.conf.Configuration;
>
> public class FixConfigFile {
>
> public static final String INDEX_WAL_EDIT_CODEC_CLASS_NAME =
> "org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec";
>  public static final String WAL_EDIT_CODEC_CLASS_KEY =
> "org.apache.hadoop.hbase.regionserver.wal.codec";
>  public static void main(String[] args) {
> Configuration config=new Configuration();
> isWALEditCodecSet(config);
>
> }
> public static boolean isWALEditCodecSet(Configuration conf) {
> // check to see if the WALEditCodec is installed
> try {
> // Use reflection to load the IndexedWALEditCodec, since it
> may not load with an older version
> // of HBase
> Class.forName(INDEX_WAL_EDIT_CODEC_CLASS_NAME);
> System.out.println("Found");
> } catch (Throwable t) {
> System.out.println("Error");
> return false;
> }
> if
> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
> null))) {
> // its installed, and it can handle compression and
> non-compression cases
> System.out.println("True");
> return true;
> }
> System.out.println("Not Found");
> return false;
> }
>
> }
> 
>
> am not sure this is how you want me to execute the code...If am wrong
> please guide me...
>
>
>
> On Sat, Aug 9, 2014 at 8:32 PM, Jesse Yates 
> wrote:
>
>> When you run
>>$ bin/hbase classpath
>> What do you get? Should help illuminate if everything is setup right.
>>
>> If the phoenix jar is there, then check the contents of the jar (
>> http://docs.oracle.com/javase/tutorial/deployment/jar/view.html) and
>> make sure the classes are present.
>>  On Aug 9, 2014 1:03 AM, "Saravanan A"  wrote:
>>
>>> Hi Jesse,
>>>
>>> I ran the following code to test the existence of the classes you asked
>>> me to check. I initialized the two constants to the following values.
>>>
>>> ===
>>> public static final String INDEX_WAL_EDIT_CODEC_CLASS_NAME =
>>> "org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec";
>>>
>>> public static final String WAL_EDIT_CODEC_CLASS_KEY =
>>> "hbase.regionserver.wal.codec";
>>> ==
>>>
>>> Then I ran the following code and got the error "Not found" in the
>>> equality test.
>>>
>>> 
>>> if
>>> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
>>> null))) {
>>> // its installed, and it can handle compression and
>>> non-compression cases
>>> System.out.println("True");
>>> return true;
>>> }
>>> System.out.println("Not Found");
>>> 
>>>
>>> I am not sure, if I initialized the values for the constants correctly.
>>> If I did, then I think some jars are missing or I have incorrect version.
>>> We use CDH 4.7 which has the Hbase version of 0.94.15 and Phoenix
>>> version of 3.0
>>>
>>> Can you tell me how to make this work? Your assistance is greatly
>>> appreciated.
>>>
>>> Regards,
>>> Saravanan.A
>>>
>>> Full code
>>> ==
>>> public static void main(String[] args) {
>>> Configuration config=new Configuration();
>>> isWALEditCodecSet(config);
>>>
>>> }
>>> public static boolean isWALEditCodecSet(Configuration conf) {
>>> // check to see if the WALEditCodec is installed
>>> tr

Re: Error when try to create mutable secondary index...

[Jesse Yates]

When you run
   $ bin/hbase classpath
What do you get? Should help illuminate if everything is setup right.

If the phoenix jar is there, then check the contents of the jar (
http://docs.oracle.com/javase/tutorial/deployment/jar/view.html) and make
sure the classes are present.
 On Aug 9, 2014 1:03 AM, "Saravanan A"  wrote:

> Hi Jesse,
>
> I ran the following code to test the existence of the classes you asked me
> to check. I initialized the two constants to the following values.
>
> ===
> public static final String INDEX_WAL_EDIT_CODEC_CLASS_NAME =
> "org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec";
>
> public static final String WAL_EDIT_CODEC_CLASS_KEY =
> "hbase.regionserver.wal.codec";
> ==
>
> Then I ran the following code and got the error "Not found" in the
> equality test.
>
> 
> if
> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
> null))) {
> // its installed, and it can handle compression and
> non-compression cases
> System.out.println("True");
> return true;
> }
> System.out.println("Not Found");
> 
>
> I am not sure, if I initialized the values for the constants correctly. If
> I did, then I think some jars are missing or I have incorrect version.
> We use CDH 4.7 which has the Hbase version of 0.94.15 and Phoenix version
> of 3.0
>
> Can you tell me how to make this work? Your assistance is greatly
> appreciated.
>
> Regards,
> Saravanan.A
>
> Full code
> ==
> public static void main(String[] args) {
> Configuration config=new Configuration();
> isWALEditCodecSet(config);
>
> }
> public static boolean isWALEditCodecSet(Configuration conf) {
> // check to see if the WALEditCodec is installed
> try {
> // Use reflection to load the IndexedWALEditCodec, since it
> may not load with an older version
> // of HBase
> Class.forName(INDEX_WAL_EDIT_CODEC_CLASS_NAME);
> System.out.println("Found");
> } catch (Throwable t) {
> System.out.println("Error");
> return false;
> }
> if
> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
> null))) {
> // its installed, and it can handle compression and
> non-compression cases
> System.out.println("True");
> return true;
> }
> System.out.println("Not Found");
> return false;
> }
>
>
>
> On Sat, Aug 9, 2014 at 12:02 AM, Jesse Yates 
> wrote:
>
>> This error is thrown when on the server-side, the following code returns
>> false (IndexManagementUtil#isWALEditCodecSet):
>>
>> public static boolean isWALEditCodecSet(Configuration conf) {
>>> // check to see if the WALEditCodec is installed
>>> try {
>>> // Use reflection to load the IndexedWALEditCodec, since it
>>> may not load with an older version
>>> // of HBase
>>> Class.forName(INDEX_WAL_EDIT_CODEC_CLASS_NAME);
>>> } catch (Throwable t) {
>>> return false;
>>> }
>>> if
>>> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
>>> null))) {
>>> // its installed, and it can handle compression and
>>> non-compression cases
>>> return true;
>>> }
>>> return false;
>>> }
>>>
>>
>>  You could just put this into a main method in a java class, put that in
>> the classpath of your HBase install on one of the machines on your cluster
>> and run it from the HBase command line to make sure that it passes.
>> Otherwise, you might not have the actual right configs (copy-paste error?)
>> or missing the right jars.
>>
>>
>> Also, FWIW, this property:
>>
>>  
>>>  hbase.region.server.rpc.scheduler.factory.class
>>>
>>>  
>>> org.apache.phoenix.hbase.index.ipc.PhoenixIndexRpcSchedulerFactory
>>>  Factory to create the Phoenix RPC Scheduler that knows
>>> to put index updates into index queues
>>>
>> 
>>>
>>
>>  is only valid in HBase 0.98.4+ (as pointed out in the section "Advanced
>> Setup - Removing Index Deadlocks (0.98.4+)"). However, it should still be
>> fine to have in older versions.
>>
>>
>>
>>

Re: Error when try to create mutable secondary index...

[Jesse Yates]

This error is thrown when on the server-side, the following code returns
false (IndexManagementUtil#isWALEditCodecSet):

public static boolean isWALEditCodecSet(Configuration conf) {
> // check to see if the WALEditCodec is installed
> try {
> // Use reflection to load the IndexedWALEditCodec, since it
> may not load with an older version
> // of HBase
> Class.forName(INDEX_WAL_EDIT_CODEC_CLASS_NAME);
> } catch (Throwable t) {
> return false;
> }
> if
> (INDEX_WAL_EDIT_CODEC_CLASS_NAME.equals(conf.get(WAL_EDIT_CODEC_CLASS_KEY,
> null))) {
> // its installed, and it can handle compression and
> non-compression cases
> return true;
> }
> return false;
> }
>

 You could just put this into a main method in a java class, put that in
the classpath of your HBase install on one of the machines on your cluster
and run it from the HBase command line to make sure that it passes.
Otherwise, you might not have the actual right configs (copy-paste error?)
or missing the right jars.


Also, FWIW, this property:

 
>  hbase.region.server.rpc.scheduler.factory.class
>
>  
> org.apache.phoenix.hbase.index.ipc.PhoenixIndexRpcSchedulerFactory
>  Factory to create the Phoenix RPC Scheduler that knows
> to put index updates into index queues
>

>

 is only valid in HBase 0.98.4+ (as pointed out in the section "Advanced
Setup - Removing Index Deadlocks (0.98.4+)"). However, it should still be
fine to have in older versions.




---
Jesse Yates
@jesse_yates
jyates.github.com


On Fri, Aug 8, 2014 at 2:18 AM, Saravanan A 
wrote:

> This is my Hbase-site.xml file...
>
>
> 
> 
> 
>
>  
>  hbase.regionserver.wal.codec
>
>  org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
>  
>  
>  hbase.region.server.rpc.scheduler.factory.class
>
>  
> org.apache.phoenix.hbase.index.ipc.PhoenixIndexRpcSchedulerFactory
>  Factory to create the Phoenix RPC Scheduler that knows
> to put index updates into index queues
>  
>
>   
> hbase.rootdir
> hdfs://alpmas.alp.com:8020/hbase
>   
>   
> hbase.client.write.buffer
> 2097152
>   
>   
> hbase.client.pause
> 1000
>   
>   
> hbase.client.retries.number
> 10
>   
>   
> hbase.client.scanner.caching
> 1000
>   
>   
> hbase.client.keyvalue.maxsize
> 20971520
>   
>   
> hbase.rpc.timeout
> 120
>   
>   
> hbase.security.authentication
> simple
>   
>   
> zookeeper.session.timeout
> 24
>   
>   
> zookeeper.retries
> 5
>   
>   
> zookeeper.pause
> 5000
>   
>   
> zookeeper.znode.parent
> /hbase
>   
>   
> zookeeper.znode.rootserver
> root-region-server
>   
>   
> hbase.zookeeper.quorum
> zk3.alp.com,zk2.alp.com,zk1.alp.com
>   
>   
> hbase.zookeeper.property.clientPort
> 2181
>   
> 
>
>
>
> On Fri, Aug 8, 2014 at 2:46 PM, Saravanan A 
> wrote:
>
>> I already included this property in hbase-site.xml in all region
>> servers..but still am getting that error...If i define my view as
>> IMMUTABLE_ROWS = true, then i can able to create view..but i want to create
>> index for mutable..
>>
>>
>> On Fri, Aug 8, 2014 at 2:10 PM, Abhilash L L 
>> wrote:
>>
>>> Really sorry, shared the wrong config
>>>
>>>
>>>
>>> 
>>>   hbase.regionserver.wal.codec
>>>   
>>> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
>>> 
>>>
>>>
>>>
>>> Regards,
>>> Abhilash L L
>>> Capillary Technologies
>>> M:919886208262
>>> abhil...@capillarytech.com | www.capillarytech.com
>>>
>>> Email from people at capillarytech.com may not represent official
>>> policy of  Capillary Technologies unless explicitly stated. Please see our
>>> Corporate-Email-Policy
>>> <http://support.capillary.co.in/policy-public/Corporate-Email-Policy.pdf>
>>> for details. Contents of this email are confidential. Please contact the
>>> Sender if you have received this email in error.
>>>
>>>
>>>
>>> On Fri, Aug 8, 2014 at 1:07 PM, Saravanan A 
>>> wrote:
>>>
>>>> Hi Abhilash,
>>>>
>>>> Thanks for the replay...i included above property and restarted the
>>>> region servers but still am g

Re: monitoring

[Jesse Yates]

It just uses the standard phoenix connection to do the writes, so whatever
phoenix can do, it can also do.

---
Jesse Yates
@jesse_yates
jyates.github.com


On Wed, Jul 2, 2014 at 12:32 PM, Rob Anderson 
wrote:

> This is great!  As a recovering Oracle DBA, I long for the sweet taste of
> performance monitoring in Phoenix!
>
> Will the tool be able to connect to a secure hbase cluster via Kerberos
> authentication (fingers crossed)?!
>
> Rob
>
>
> On Tue, Jul 1, 2014 at 4:39 PM, Jeffrey Zhong 
> wrote:
>
>>
>> There is a plan to drop hadoop1(
>> https://issues.apache.org/jira/browse/PHOENIX-103). So only supporting
>> hadoop2 is good enough.
>>
>> -Jeffrey
>>
>> From: James Taylor 
>> Reply-To: 
>> Date: Tuesday, July 1, 2014 11:47 AM
>> To: "user@phoenix.apache.org" 
>> Subject: Re: monitoring
>>
>> +2 Awesome job, Jesse!
>>
>> On Tuesday, July 1, 2014, Eli Levine  wrote:
>>
>>> +1 for only supporting Hadoop2 initially (great work, Jesse!). It's
>>> important that Phoenix started implementing monitoring and metrics
>>> features. An incremental approach is the way to go IMHO.
>>>
>>> Thanks,
>>>
>>> Eli
>>>
>>>
>>> On Tue, Jul 1, 2014 at 11:13 AM, James Taylor 
>>> wrote:
>>>
>>>> Seems like an excellent feature to have in 4.1 which I'm hoping we can
>>>> do by end of the month. I'd rather the feature make it in and only
>>>> support hadoop2 than have it not make the release.
>>>>
>>>> Any one else have thoughts on this?
>>>>
>>>> On Tue, Jul 1, 2014 at 8:01 PM, Jesse Yates 
>>>> wrote:
>>>> > That's certainly possible, though somewhat unsatisfying :) Its not
>>>> terribly
>>>> > difficult to the Hadoop1 code - I've modularized a majority of the
>>>> code so
>>>> > all that is really needed is replicating the reading of the Hadoop1
>>>> metrics2
>>>> > classes to something the phoenix writer understands (just like in the
>>>> > Hadoop2 impl).
>>>> >
>>>> > Happy to push the code somewhere so people can take a look... or they
>>>> just
>>>> > wait a couple weeks :)
>>>> >
>>>> > ---
>>>> > Jesse Yates
>>>> > @jesse_yates
>>>> > jyates.github.com
>>>> >
>>>> >
>>>> > On Tue, Jul 1, 2014 at 10:53 AM, James Taylor >>> >
>>>> > wrote:
>>>> >>
>>>> >> How about if we make it a hadoop2 only feature?
>>>> >>
>>>> >>
>>>> >> On Tuesday, July 1, 2014, Jesse Yates 
>>>> wrote:
>>>> >>>
>>>> >>> I was working on a patch to support using Cloudera's HTrace
>>>> >>> (https://github.com/cloudera/htrace) library for phoenix queries.
>>>> There was
>>>> >>> a preliminary patch for the 2.X series, but it never got committed.
>>>> Two
>>>> >>> weeks ago I almost finished porting it to the 4.X line, but haven't
>>>> finished
>>>> >>> the piece that will support Hadoop1*. I can't seem to find the JIRA
>>>> right
>>>> >>> now though...
>>>> >>>
>>>> >>> I've gotten sidetracked on some internal work, so probably not
>>>> going to
>>>> >>> have time to wrap it up for a couple weeks.
>>>> >>>
>>>> >>> -J
>>>> >>>
>>>> >>> * It uses the Hadoop metrics2 library to collect the traces and
>>>> then has
>>>> >>> a default implementation (as a metrics sink) that writes them to a
>>>> phoenix
>>>> >>> table so it can be analyzed later. Because of the changes between
>>>> Hadoop1
>>>> >>> and 2, there is some reflection funkiness that is necessary to
>>>> support both,
>>>> >>> but just haven't finished the hadoop1 side of it.
>>>> >>> ---
>>>> >>> Jesse Yates
>>>> >>> @jesse_yates
>>>> >>> jyates.github.com
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Jul 1, 2014 at 9:29 A

Re: monitoring

[Jesse Yates]

That's certainly possible, though somewhat unsatisfying :) Its not terribly
difficult to the Hadoop1 code - I've modularized a majority of the code so
all that is really needed is replicating the reading of the Hadoop1
metrics2 classes to something the phoenix writer understands (just like in
the Hadoop2 impl).

Happy to push the code somewhere so people can take a look... or they just
wait a couple weeks :)

---
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Jul 1, 2014 at 10:53 AM, James Taylor 
wrote:

> How about if we make it a hadoop2 only feature?
>
>
> On Tuesday, July 1, 2014, Jesse Yates  wrote:
>
>> I was working on a patch to support using Cloudera's HTrace (
>> https://github.com/cloudera/htrace) library for phoenix queries. There
>> was a preliminary patch for the 2.X series, but it never got committed. Two
>> weeks ago I almost finished porting it to the 4.X line, but haven't
>> finished the piece that will support Hadoop1*. I can't seem to find the
>> JIRA right now though...
>>
>> I've gotten sidetracked on some internal work, so probably not going to
>> have time to wrap it up for a couple weeks.
>>
>> -J
>>
>> * It uses the Hadoop metrics2 library to collect the traces and then has
>> a default implementation (as a metrics sink) that writes them to a phoenix
>> table so it can be analyzed later. Because of the changes between Hadoop1
>> and 2, there is some reflection funkiness that is necessary to support
>> both, but just haven't finished the hadoop1 side of it.
>> ---
>> Jesse Yates
>> @jesse_yates
>> jyates.github.com
>>
>>
>> On Tue, Jul 1, 2014 at 9:29 AM, Stephen Sprague 
>> wrote:
>>
>>> following up to my own message i did see reference to something called
>>> "Hannibal" (https://github.com/sentric/hannibal) in the Phoenix doc.
>>> Looks like this helps with identifying imbalance within your cluster.
>>>
>>> I'll keep searching around though - a Phoenix log file looks like
>>> perhaps the best way to monitor phoenix queries at this point as i don't
>>> think there is any self-aware queries to ask of it.  That's a jdbc driver
>>> issue anyway - not phoenix.
>>>
>>> ​Cheers,
>>> Stephen.​
>>>
>>>
>>>
>>> On Mon, Jun 30, 2014 at 10:17 PM, Stephen Sprague 
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> May i ask what tools you use to monitor (active) Phoenix queries?
>>>>
>>>> looking at the web ui for hbase master and region servers it isn't
>>>> exactly intuitive as to what *phoenix* is doing.  Its probably a stretch to
>>>> ask if there's something like "show processlist" (ala mysql) out there
>>>> somewhere.
>>>>
>>>> so is there a log file with metrics in it that perhaps i could tail or
>>>> analyze in realtime?   perhaps even feed it back into hbase? :)  hey, wait
>>>> a sec, anything in those system tables that is realtime-ish?
>>>>
>>>> regardless, kinda curious how people monitor the Phoenix plant.
>>>>
>>>> thanks,
>>>> Stephen.
>>>>
>>>>
>>>>
>>>
>>

Re: monitoring

[Jesse Yates]

I was working on a patch to support using Cloudera's HTrace (
https://github.com/cloudera/htrace) library for phoenix queries. There was
a preliminary patch for the 2.X series, but it never got committed. Two
weeks ago I almost finished porting it to the 4.X line, but haven't
finished the piece that will support Hadoop1*. I can't seem to find the
JIRA right now though...

I've gotten sidetracked on some internal work, so probably not going to
have time to wrap it up for a couple weeks.

-J

* It uses the Hadoop metrics2 library to collect the traces and then has a
default implementation (as a metrics sink) that writes them to a phoenix
table so it can be analyzed later. Because of the changes between Hadoop1
and 2, there is some reflection funkiness that is necessary to support
both, but just haven't finished the hadoop1 side of it.
---
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Jul 1, 2014 at 9:29 AM, Stephen Sprague  wrote:

> following up to my own message i did see reference to something called
> "Hannibal" (https://github.com/sentric/hannibal) in the Phoenix doc.
> Looks like this helps with identifying imbalance within your cluster.
>
> I'll keep searching around though - a Phoenix log file looks like perhaps
> the best way to monitor phoenix queries at this point as i don't think
> there is any self-aware queries to ask of it.  That's a jdbc driver issue
> anyway - not phoenix.
>
> ​Cheers,
> Stephen.​
>
>
>
> On Mon, Jun 30, 2014 at 10:17 PM, Stephen Sprague 
> wrote:
>
>> Hi guys,
>>
>> May i ask what tools you use to monitor (active) Phoenix queries?
>>
>> looking at the web ui for hbase master and region servers it isn't
>> exactly intuitive as to what *phoenix* is doing.  Its probably a stretch to
>> ask if there's something like "show processlist" (ala mysql) out there
>> somewhere.
>>
>> so is there a log file with metrics in it that perhaps i could tail or
>> analyze in realtime?   perhaps even feed it back into hbase? :)  hey, wait
>> a sec, anything in those system tables that is realtime-ish?
>>
>> regardless, kinda curious how people monitor the Phoenix plant.
>>
>> thanks,
>> Stephen.
>>
>>
>>
>