date:20161022

PhoenixIOException: Table 'unionSchemaName.unionTableName' was not found

2016-10-22 Thread Parveen Jain

While running this query from spark phoenix connector:


select distinct(C_TXN.CUSTMR_ID) from CUS_TXN where (CUS_TXN.TXN_TYPE='') 
and (substr(CUS_TXN.ROW_KEY,0,8)>='20160101') group by CUS_TXN.CUSTMR_ID having 
sum(CUS_TXN.TXN_AMOUNT)>=300 union all select distinct(CUS_TXN.CUSTMR_ID) from 
CUS_TXN where (CUS_TXN.TXN_TYPE='') and 
(substr(CUS_TXN.ROW_KEY,0,8)>='20160101') group by CUS_TXN.CUSTMR_ID having 
sum(CUS_TXN.TXN_AMOUNT)>100



getting below exception:

Getting some phoenix exception for below query:
java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: 
Table 'unionSchemaName.unionTableName' was not found, got: hbase:namespace.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(Phoen



my code for fetching records is:

PhoenixConfigurationUtil.setInputTableName(configuration , TABLE_NAME);
PhoenixConfigurationUtil.setOutputTableName(configuration ,TABLE_NAME);
PhoenixConfigurationUtil.setInputQuery(configuration, QueryToRun);
PhoenixConfigurationUtil.setInputClass(configuration, DataRecord.class);

configuration.setClass(JobContext.OUTPUT_FORMAT_CLASS_ATTR,PhoenixOutputFormat.class,
 OutputFormat.class);

@SuppressWarnings("unchecked")
JavaPairRDD stocksRDD = jsc.newAPIHadoopRDD(
configuration,
PhoenixInputFormat.class,
NullWritable.class,
DataRecord.class);


Regards,
Parveen Jain




Any pointer why this could be happening.


Regards,

Parveen Jain

Re: Index in Phoenix view on Hbase is not updated

2016-10-22 Thread Mich Talebzadeh

A workout I deployed was to rebuild the index immediately after bulk load
of data into Hbase table

ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 22 October 2016 at 18:25, Mich Talebzadeh 
wrote:

> Sorry Ted,
>
> This is the syntax for view
>
> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
> "price_info"."price" VARCHAR);
>
> Thanks James for clarification.
>
> My understanding is that when one creates an index on a Phoenix view on an
> Hbase table,  a local index file is created in Phoenix with data as is and
> that Phoenix table has no way of knowing if new data is added to Hbase
> table.
>
> From an operational point of view unless index in Phoenix is immediately
> updated after any append to Hbase, then it is pretty useless.
>
> In my case data is inserted to Hbase table. I am just using Phoenix for
> data queries (DQ) as opposed to inserts.
>
> Regards,
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 22 October 2016 at 17:59, James Taylor  wrote:
>
>> Hi Mich,
>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>> data.
>> Thanks,
>> James
>>
>>
>> On Saturday, October 22, 2016, Ted Yu  wrote:
>>
>>> The first statement creates index, not view.
>>>
>>> Can you check ?
>>>
>>> Cheers
>>>
>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have a Hbase table that is populated via
>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>> > through bulk load ever 15 minutes. This works fine.
>>> >
>>> > In Phoenix I created a view on this table
>>> >
>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>> > "price_info"."timecreated");
>>> >
>>> > This also does what is supposed to do and shows correct count.
>>> >
>>> > I then created an index in Phoenix as below
>>> >
>>> > create index index_dx1 on "marketDataHbase"
>>> > ("price_info"."timecreated","price_info"."ticker",
>>> "price_info"."price");
>>> >
>>> > that showed the records OK at that time. I verified this using explain
>>> >
>>> >
>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>> > "marketDataHbase";
>>> > +-+
>>> > |  PLAN   |
>>> > +-+
>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1  |
>>> > | SERVER FILTER BY FIRST KEY ONLY |
>>> > | SERVER AGGREGATE INTO SINGLE ROW|
>>> > +-+
>>> >
>>> > Now the issue is that the above does not show new data since build in
>>> Hbase
>>> > table unless I do the following:
>>> >
>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>> "marketDataHbase"
>>> > rebuild;
>>> >
>>> >
>>> > Which is not what an index should do (The covered index should be
>>> > maintained automatically).
>>> > The simple issue is how to overcome this problem?
>>> >
>>> > As I understand the index in Phoenix ia another file independent of the
>>> > original phoenix view so I assume that this index file is not updated
>>> for
>>> > one reason or other?
>>> >
>>> > Thanks
>>>
>>
>

Re: Index in Phoenix view on Hbase is not updated

2016-10-22 Thread Mich Talebzadeh

Sorry Ted,

This is the syntax for view

create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
“price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
"price_info"."price" VARCHAR);

Thanks James for clarification.

My understanding is that when one creates an index on a Phoenix view on an
Hbase table,  a local index file is created in Phoenix with data as is and
that Phoenix table has no way of knowing if new data is added to Hbase
table.

>From an operational point of view unless index in Phoenix is immediately
updated after any append to Hbase, then it is pretty useless.

In my case data is inserted to Hbase table. I am just using Phoenix for
data queries (DQ) as opposed to inserts.

Regards,







Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 22 October 2016 at 17:59, James Taylor  wrote:

> Hi Mich,
> Phoenix indexes are only updated if you use Phoenix APIs to input the data.
> Thanks,
> James
>
>
> On Saturday, October 22, 2016, Ted Yu  wrote:
>
>> The first statement creates index, not view.
>>
>> Can you check ?
>>
>> Cheers
>>
>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh 
>> wrote:
>> >
>> > Hi,
>> >
>> > I have a Hbase table that is populated via
>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>> > through bulk load ever 15 minutes. This works fine.
>> >
>> > In Phoenix I created a view on this table
>> >
>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>> > "price_info"."timecreated");
>> >
>> > This also does what is supposed to do and shows correct count.
>> >
>> > I then created an index in Phoenix as below
>> >
>> > create index index_dx1 on "marketDataHbase"
>> > ("price_info"."timecreated","price_info"."ticker",
>> "price_info"."price");
>> >
>> > that showed the records OK at that time. I verified this using explain
>> >
>> >
>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>> > "marketDataHbase";
>> > +-+
>> > |  PLAN   |
>> > +-+
>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1  |
>> > | SERVER FILTER BY FIRST KEY ONLY |
>> > | SERVER AGGREGATE INTO SINGLE ROW|
>> > +-+
>> >
>> > Now the issue is that the above does not show new data since build in
>> Hbase
>> > table unless I do the following:
>> >
>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
>> > rebuild;
>> >
>> >
>> > Which is not what an index should do (The covered index should be
>> > maintained automatically).
>> > The simple issue is how to overcome this problem?
>> >
>> > As I understand the index in Phoenix ia another file independent of the
>> > original phoenix view so I assume that this index file is not updated
>> for
>> > one reason or other?
>> >
>> > Thanks
>>
>

Re: Index in Phoenix view on Hbase is not updated

2016-10-22 Thread James Taylor

Hi Mich,
Phoenix indexes are only updated if you use Phoenix APIs to input the data.
Thanks,
James

On Saturday, October 22, 2016, Ted Yu  wrote:

> The first statement creates index, not view.
>
> Can you check ?
>
> Cheers
>
> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh  > wrote:
> >
> > Hi,
> >
> > I have a Hbase table that is populated via
> > org.apache.hadoop.hbase.mapreduce.ImportTsv
> > through bulk load ever 15 minutes. This works fine.
> >
> > In Phoenix I created a view on this table
> >
> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
> > "price_info"."timecreated");
> >
> > This also does what is supposed to do and shows correct count.
> >
> > I then created an index in Phoenix as below
> >
> > create index index_dx1 on "marketDataHbase"
> > ("price_info"."timecreated","price_info"."ticker",
> "price_info"."price");
> >
> > that showed the records OK at that time. I verified this using explain
> >
> >
> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
> > "marketDataHbase";
> > +-+
> > |  PLAN   |
> > +-+
> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1  |
> > | SERVER FILTER BY FIRST KEY ONLY |
> > | SERVER AGGREGATE INTO SINGLE ROW|
> > +-+
> >
> > Now the issue is that the above does not show new data since build in
> Hbase
> > table unless I do the following:
> >
> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
> > rebuild;
> >
> >
> > Which is not what an index should do (The covered index should be
> > maintained automatically).
> > The simple issue is how to overcome this problem?
> >
> > As I understand the index in Phoenix ia another file independent of the
> > original phoenix view so I assume that this index file is not updated for
> > one reason or other?
> >
> > Thanks
>

Re: Index in Phoenix view on Hbase is not updated

2016-10-22 Thread Ted Yu

The first statement creates index, not view. 

Can you check ?

Cheers

> On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh  
> wrote:
> 
> Hi,
> 
> I have a Hbase table that is populated via
> org.apache.hadoop.hbase.mapreduce.ImportTsv
> through bulk load ever 15 minutes. This works fine.
> 
> In Phoenix I created a view on this table
> 
> jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
> "marketDataHbase" ("price_info"."ticker", "price_info"."price",
> "price_info"."timecreated");
> 
> This also does what is supposed to do and shows correct count.
> 
> I then created an index in Phoenix as below
> 
> create index index_dx1 on "marketDataHbase"
> ("price_info"."timecreated","price_info"."ticker", "price_info"."price");
> 
> that showed the records OK at that time. I verified this using explain
> 
> 
> 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
> "marketDataHbase";
> +-+
> |  PLAN   |
> +-+
> | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1  |
> | SERVER FILTER BY FIRST KEY ONLY |
> | SERVER AGGREGATE INTO SINGLE ROW|
> +-+
> 
> Now the issue is that the above does not show new data since build in Hbase
> table unless I do the following:
> 
> 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
> rebuild;
> 
> 
> Which is not what an index should do (The covered index should be
> maintained automatically).
> The simple issue is how to overcome this problem?
> 
> As I understand the index in Phoenix ia another file independent of the
> original phoenix view so I assume that this index file is not updated for
> one reason or other?
> 
> Thanks

Index in Phoenix view on Hbase is not updated

2016-10-22 Thread Mich Talebzadeh

Hi,

I have a Hbase table that is populated via
org.apache.hadoop.hbase.mapreduce.ImportTsv
through bulk load ever 15 minutes. This works fine.

In Phoenix I created a view on this table

jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
"marketDataHbase" ("price_info"."ticker", "price_info"."price",
"price_info"."timecreated");

This also does what is supposed to do and shows correct count.

I then created an index in Phoenix as below

create index index_dx1 on "marketDataHbase"
("price_info"."timecreated","price_info"."ticker", "price_info"."price");

that showed the records OK at that time. I verified this using explain


0: jdbc:phoenix:rhes564:2181> explain select count(1) from
"marketDataHbase";
+-+
|  PLAN   |
+-+
| CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1  |
| SERVER FILTER BY FIRST KEY ONLY |
| SERVER AGGREGATE INTO SINGLE ROW|
+-+

Now the issue is that the above does not show new data since build in Hbase
table unless I do the following:

0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
rebuild;


Which is not what an index should do (The covered index should be
maintained automatically).
The simple issue is how to overcome this problem?

As I understand the index in Phoenix ia another file independent of the
original phoenix view so I assume that this index file is not updated for
one reason or other?

Thanks

Re: [jira] Vivek Paranthaman shared "PHOENIX-3395: ResultSet .next() throws commons-io exception" with you

2016-10-22 Thread Vivek Paranthaman

If you look into it,
DeferredFileOutputStream.(ILjava/lang/String;
Ljava/lang/String;Ljava/io/File;)
is seeking for 3 parameterised method, which is not available in the Api -
https://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/output/DeferredFileOutputStream.html

On Sat, Oct 22, 2016 at 1:15 PM, Ciureanu Constantin <
ciureanu.constan...@gmail.com> wrote:

> Not sure what ti say, check your apache-commons version, perhaps it's
> picking an older one in the classpath.
>
> În vin., 21 oct. 2016, 09:36 Vivek Paranthaman (JIRA), 
> a scris:
>
>> Vivek Paranthaman shared an issue with you
>> 
>>
>>
>>
>> > ResultSet .next() throws commons-io exception
>> > -
>> >
>> > Key: PHOENIX-3395
>> > URL: https://issues.apache.org/jira/browse/PHOENIX-3395
>> > Project: Phoenix
>> >  Issue Type: Bug
>> >Affects Versions: 4.5.1
>> > Environment: java -version: 1.7
>> >Reporter: Vivek Paranthaman
>> >
>> > Exception in thread "main" org.apache.phoenix.exception.PhoenixIOException:
>> java.lang.NoSuchMethodError: org.apache.commons.io.output.D
>> eferredFileOutputStream.(ILjava/lang/String;Ljava/
>> lang/String;Ljava/io/File;)V
>> >   at org.apache.phoenix.util.ServerUtil.parseServerException(Serv
>> erUtil.java:108)
>> >   at org.apache.phoenix.iterate.BaseResultIterators.getIterators(
>> BaseResultIterators.java:553)
>> >   at org.apache.phoenix.iterate.MergeSortResultIterator.getIterat
>> ors(MergeSortResultIterator.java:48)
>> >   at org.apache.phoenix.iterate.MergeSortResultIterator.minIterat
>> or(MergeSortResultIterator.java:84)
>> >   at org.apache.phoenix.iterate.MergeSortResultIterator.next(Merg
>> eSortResultIterator.java:111)
>> >   at org.apache.phoenix.iterate.MergeSortTopNResultIterator.next(
>> MergeSortTopNResultIterator.java:85)
>> >   at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultS
>> et.java:773)
>>
>>  Also shared with
>>   jamestaylor
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>
>

Re: Creating Covering index on Phoenix

2016-10-22 Thread Mich Talebzadeh

Thanks Sergey,

In this case the phoenix view is defined on Hbase table.

Hbase table is updated every 15 minutes via cron that uses
org.apache.hadoop.hbase.mapreduce.ImportTsv  to bulk load data into Hbase
table,

So if I create index on my view in Phoenix, will that index be maintained?

regards

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 21 October 2016 at 23:35, Sergey Soldatov 
wrote:

> Hi Mich,
>
> It's really depends on the query that you are going to use. If conditions
> will be applied only by time column you may create index like
> create index I on "marketDataHbase" ("timecreated") include ("ticker",
> "price");
> If the conditions will be applied on others columns as well, you may use
> create index I on "marketDataHbase" ("timecreated","ticker", "price");
>
> Index is updated together with the user table if you are using phoenix
> jdbc driver or phoenix bulk load tools to ingest the data.
>
> Thanks,
> Sergey
>
> On Fri, Oct 21, 2016 at 4:43 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>>
>>
>> Hi,
>>
>> I have  a Phoenix table on Hbase as follows:
>>
>> [image: Inline images 1]
>>
>> I want to create a covered index to cover the three columns: ticker,
>> timecreated, price
>>
>> More importantly I want the index to be maintained when new rows are
>> added to Hbase table.
>>
>> What is the best way of achieving this?
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
>

Re: [jira] Vivek Paranthaman shared "PHOENIX-3395: ResultSet .next() throws commons-io exception" with you

2016-10-22 Thread Ciureanu Constantin

Not sure what ti say, check your apache-commons version, perhaps it's
picking an older one in the classpath.

În vin., 21 oct. 2016, 09:36 Vivek Paranthaman (JIRA),  a
scris:

> Vivek Paranthaman shared an issue with you
> 
>
>
>
> > ResultSet .next() throws commons-io exception
> > -
> >
> > Key: PHOENIX-3395
> > URL: https://issues.apache.org/jira/browse/PHOENIX-3395
> > Project: Phoenix
> >  Issue Type: Bug
> >Affects Versions: 4.5.1
> > Environment: java -version: 1.7
> >Reporter: Vivek Paranthaman
> >
> > Exception in thread "main"
> org.apache.phoenix.exception.PhoenixIOException:
> java.lang.NoSuchMethodError:
> org.apache.commons.io.output.DeferredFileOutputStream.(ILjava/lang/String;Ljava/lang/String;Ljava/io/File;)V
> >   at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
> >   at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:553)
> >   at
> org.apache.phoenix.iterate.MergeSortResultIterator.getIterators(MergeSortResultIterator.java:48)
> >   at
> org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:84)
> >   at
> org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:111)
> >   at
> org.apache.phoenix.iterate.MergeSortTopNResultIterator.next(MergeSortTopNResultIterator.java:85)
> >   at
> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:773)
>
>  Also shared with
>   jamestaylor
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

PhoenixIOException: Table 'unionSchemaName.unionTableName' was not found

Re: Index in Phoenix view on Hbase is not updated

Re: Index in Phoenix view on Hbase is not updated

Re: Index in Phoenix view on Hbase is not updated

Re: Index in Phoenix view on Hbase is not updated

Index in Phoenix view on Hbase is not updated

Re: [jira] Vivek Paranthaman shared "PHOENIX-3395: ResultSet .next() throws commons-io exception" with you

Re: Creating Covering index on Phoenix

Re: [jira] Vivek Paranthaman shared "PHOENIX-3395: ResultSet .next() throws commons-io exception" with you

9 matches

Site Navigation

Mail list logo

Footer information