Re: Phoenix and NodeJS
Have you looked at paging [1] using Phoenix's row-value constructors together with the LIMIT clause? That might be what you are looking for. [1] http://phoenix.apache.org/paged.html Eli On Mon, May 18, 2015 at 6:46 AM, Isart Montane isart.mont...@gmail.com wrote: Hi, the company I work for is performing some tests on Phoenix with NodeJS. For simple queries I didn't had any problem, but as soon as I start to use our app I'm getting process out of memory errors on the client when I runs queries that return a big number of rows (i.e. 400k) . I think the problem is that the client tries to buffer all the results in RAM and that kills it. The same query runs fine when I run it with sqline. So, is there a way to tell the client to stream the results (or batch them) instead of buffering them all? is raising the client memory the only solution? I'm using phoenix-4.3.1 and https://github.com/gaodazhu/phoenix-client as the NodeJS driver Thanks, Isart Montane
Re: Phoenix and NodeJS
Hi Eli, thanks a lot for your answer. That might be a workaround but I was hoping to get a more generic answer I can apply to the driver/phoenix since that will require me lots of changes to the code. Any clue on why it works with sqline but not trough the node driver? On Mon, May 18, 2015 at 4:20 PM, Eli Levine elilev...@gmail.com wrote: Have you looked at paging [1] using Phoenix's row-value constructors together with the LIMIT clause? That might be what you are looking for. [1] http://phoenix.apache.org/paged.html Eli On Mon, May 18, 2015 at 6:46 AM, Isart Montane isart.mont...@gmail.com wrote: Hi, the company I work for is performing some tests on Phoenix with NodeJS. For simple queries I didn't had any problem, but as soon as I start to use our app I'm getting process out of memory errors on the client when I runs queries that return a big number of rows (i.e. 400k) . I think the problem is that the client tries to buffer all the results in RAM and that kills it. The same query runs fine when I run it with sqline. So, is there a way to tell the client to stream the results (or batch them) instead of buffering them all? is raising the client memory the only solution? I'm using phoenix-4.3.1 and https://github.com/gaodazhu/phoenix-client as the NodeJS driver Thanks, Isart Montane
Re: Problems with Phoenix and HBase
Sending to Phoenix user mailing list. Here is the thread: http://search-hadoop.com/m/YGbbu2WzHtZBkq1 On Mon, May 18, 2015 at 7:20 AM, Asfare aman...@hotmail.com wrote: Can someone give some tips? -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Problems-with-Phoenix-and-HBase-tp4071362p4071537.html Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Trying to setup unittests to query phoenix test db but getting UnsupportedOperationException
Thanks! I will look into it.
On 05/15/2015 06:24 PM, James Taylor wrote:
You'll want to derive from BaseHBaseManagedTimeIT. The
BaseConnectionlessQueryTest class is for compile-time only or negative
tests as it doesn't spin up any mini cluster.
Thanks,
James
On Fri, May 15, 2015 at 5:41 AM, Ron van der Vegt
ron.van.der.v...@openindex.io wrote:
Hello everyone,
I'm currently developing an Rest API which should query a phoenix table, and
return it in JSON. Currently have no issues with building this API, but it
would be really nice if I could write unittests with dummy data to test our
created API calls.
I was getting into the right direction, I hope, by extending the
BaseConnectionlessQueryTest class and setup a test database:
String ddl = CREATE TABLE test (id VARCHAR not null primary key, test_value
CHAR(16);
createTestTable(getUrl(), ddl, (byte[][]) null, (Long) null);
And it looks like I could also upsert some data:
Properties props = new Properties();
PhoenixConnection conn =
(PhoenixConnection)DriverManager.getConnection(jdbc:phoenix:none;test=true,
props);
PreparedStatement statement = conn.prepareStatement(UPSERT INTO test(id)
VALUES ('meh'));
statement.execute();
But when I want to select data:
PreparedStatement statement = conn.prepareStatement(SELECT * FROM test);
ResultSet rs = statement.executeQuery();
while (rs.next()) {
System.out.println(rs.getString(ID));
}
I am get an UnsupportedOperationException. Could someone please explain to
me what I'm doing wrong, or that my use case is possible somehow?
Thanks in advice!
Ron
Re: Phoenix and NodeJS
I don't have info on what your app does with results from Phoenix. If the app is constructing some sort of object representations from Phoenix results and holding on to them, I would look at what the memory footprint of that is. I know this isn't very helpful but at this point I would try to dig deeper into your app and the NodeJS driver rather than Phoenix, since you mentioned the same queries run fine in sqlline. On Mon, May 18, 2015 at 7:30 AM, Isart Montane isart.mont...@gmail.com wrote: Hi Eli, thanks a lot for your answer. That might be a workaround but I was hoping to get a more generic answer I can apply to the driver/phoenix since that will require me lots of changes to the code. Any clue on why it works with sqline but not trough the node driver? On Mon, May 18, 2015 at 4:20 PM, Eli Levine elilev...@gmail.com wrote: Have you looked at paging [1] using Phoenix's row-value constructors together with the LIMIT clause? That might be what you are looking for. [1] http://phoenix.apache.org/paged.html Eli On Mon, May 18, 2015 at 6:46 AM, Isart Montane isart.mont...@gmail.com wrote: Hi, the company I work for is performing some tests on Phoenix with NodeJS. For simple queries I didn't had any problem, but as soon as I start to use our app I'm getting process out of memory errors on the client when I runs queries that return a big number of rows (i.e. 400k) . I think the problem is that the client tries to buffer all the results in RAM and that kills it. The same query runs fine when I run it with sqline. So, is there a way to tell the client to stream the results (or batch them) instead of buffering them all? is raising the client memory the only solution? I'm using phoenix-4.3.1 and https://github.com/gaodazhu/phoenix-client as the NodeJS driver Thanks, Isart Montane
Re: Phoenix and NodeJS
Hi Isart,
That code isn't Phoenix code. This sounds like a Node JS issue. Vaclav
has done a lot with Node JS, so he may be able to give you some tips.
Thanks,
James
On Mon, May 18, 2015 at 9:06 AM, Isart Montane isart.mont...@gmail.com wrote:
Hi Eli,
thanks a lot for your comments. I think you are right. I found the client
code that's causing the issue. Do you have an example I can use to patch it?
is that the recommended way to access phoenix? I've seen on the web that
there's also a query server available, is it worth a try?
public String[] query(String sql)
{
ListString lsResults = new ArrayList();
Connection conn = null;
try
{
conn = this.dataSource.getConnection();
ResultSet rs = conn.createStatement().executeQuery(sql);
ResultSetMetaData data = rs.getMetaData();
int numberOfColumns = data.getColumnCount();
ListString lsRows = new ArrayList();
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(data.getColumnName(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
while (rs.next())
{
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(rs.getString(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
}
rs.close();
conn.close();
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
return (String[])lsResults.toArray(new String[lsResults.size()]);
}
On Mon, May 18, 2015 at 5:43 PM, Eli Levine elilev...@gmail.com wrote:
I don't have info on what your app does with results from Phoenix. If the
app is constructing some sort of object representations from Phoenix results
and holding on to them, I would look at what the memory footprint of that
is. I know this isn't very helpful but at this point I would try to dig
deeper into your app and the NodeJS driver rather than Phoenix, since you
mentioned the same queries run fine in sqlline.
On Mon, May 18, 2015 at 7:30 AM, Isart Montane isart.mont...@gmail.com
wrote:
Hi Eli,
thanks a lot for your answer. That might be a workaround but I was hoping
to get a more generic answer I can apply to the driver/phoenix since that
will require me lots of changes to the code.
Any clue on why it works with sqline but not trough the node driver?
On Mon, May 18, 2015 at 4:20 PM, Eli Levine elilev...@gmail.com wrote:
Have you looked at paging [1] using Phoenix's row-value constructors
together with the LIMIT clause? That might be what you are looking for.
[1] http://phoenix.apache.org/paged.html
Eli
On Mon, May 18, 2015 at 6:46 AM, Isart Montane isart.mont...@gmail.com
wrote:
Hi,
the company I work for is performing some tests on Phoenix with NodeJS.
For simple queries I didn't had any problem, but as soon as I start to use
our app I'm getting process out of memory errors on the client when I
runs
queries that return a big number of rows (i.e. 400k) . I think the problem
is that the client tries to buffer all the results in RAM and that kills
it.
The same query runs fine when I run it with sqline.
So, is there a way to tell the client to stream the results (or batch
them) instead of buffering them all? is raising the client memory the only
solution?
I'm using phoenix-4.3.1 and https://github.com/gaodazhu/phoenix-client
as the NodeJS driver
Thanks,
Isart Montane
Problem while upgrading from Phooenix 4.0.0 to 4.3.1
Hello, 1. Currently using phoenix 4.0.0 incubating for both client and server. 2. Upgraded to 4.3.1(most recent) 3. While trying to connect using the client in command line (using ./sqlline.py) the connection could not be success throwing the following error. 1) *Error: ERROR 1013 (42M04): Table already exists. tableName=SYSTEM.CATALOG (state=42M04,code=1013)org.apache.phoenix.schema.NewerTableAlreadyExistsException: ERROR 1013 (42M04): Table already exists. tableName=SYSTEM.CATALOG* Deleting the SYSTEM.CATALOG table works, but that is not the solution intended. What is the solution/workaround for the problem? Thanks, Arun Sabtharishi
Re: Phoenix and NodeJS
Hi Eli,
thanks a lot for your comments. I think you are right. I found the client
code that's causing the issue. Do you have an example I can use to patch
it? is that the recommended way to access phoenix? I've seen on the web
that there's also a query server available, is it worth a try?
public String[] query(String sql)
{
ListString lsResults = new ArrayList();
Connection conn = null;
try
{
conn = this.dataSource.getConnection();
ResultSet rs = conn.createStatement().executeQuery(sql);
ResultSetMetaData data = rs.getMetaData();
int numberOfColumns = data.getColumnCount();
ListString lsRows = new ArrayList();
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(data.getColumnName(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
while (rs.next())
{
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(rs.getString(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
}
rs.close();
conn.close();
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
return (String[])lsResults.toArray(new String[lsResults.size()]);
}
On Mon, May 18, 2015 at 5:43 PM, Eli Levine elilev...@gmail.com wrote:
I don't have info on what your app does with results from Phoenix. If the
app is constructing some sort of object representations from Phoenix
results and holding on to them, I would look at what the memory footprint
of that is. I know this isn't very helpful but at this point I would try to
dig deeper into your app and the NodeJS driver rather than Phoenix, since
you mentioned the same queries run fine in sqlline.
On Mon, May 18, 2015 at 7:30 AM, Isart Montane isart.mont...@gmail.com
wrote:
Hi Eli,
thanks a lot for your answer. That might be a workaround but I was hoping
to get a more generic answer I can apply to the driver/phoenix since that
will require me lots of changes to the code.
Any clue on why it works with sqline but not trough the node driver?
On Mon, May 18, 2015 at 4:20 PM, Eli Levine elilev...@gmail.com wrote:
Have you looked at paging [1] using Phoenix's row-value constructors
together with the LIMIT clause? That might be what you are looking for.
[1] http://phoenix.apache.org/paged.html
Eli
On Mon, May 18, 2015 at 6:46 AM, Isart Montane isart.mont...@gmail.com
wrote:
Hi,
the company I work for is performing some tests on Phoenix with NodeJS.
For simple queries I didn't had any problem, but as soon as I start to use
our app I'm getting process out of memory errors on the client when I
runs queries that return a big number of rows (i.e. 400k) . I think the
problem is that the client tries to buffer all the results in RAM and that
kills it. The same query runs fine when I run it with sqline.
So, is there a way to tell the client to stream the results (or batch
them) instead of buffering them all? is raising the client memory the only
solution?
I'm using phoenix-4.3.1 and https://github.com/gaodazhu/phoenix-client
as the NodeJS driver
Thanks,
Isart Montane
Re: Phoenix and NodeJS
Thanks James.
That code is from the node driver, I will try to get some advice from it's
developer.
Thanks,
On Mon, May 18, 2015 at 6:34 PM, James Taylor jamestay...@apache.org
wrote:
Hi Isart,
That code isn't Phoenix code. This sounds like a Node JS issue. Vaclav
has done a lot with Node JS, so he may be able to give you some tips.
Thanks,
James
On Mon, May 18, 2015 at 9:06 AM, Isart Montane isart.mont...@gmail.com
wrote:
Hi Eli,
thanks a lot for your comments. I think you are right. I found the client
code that's causing the issue. Do you have an example I can use to patch
it?
is that the recommended way to access phoenix? I've seen on the web that
there's also a query server available, is it worth a try?
public String[] query(String sql)
{
ListString lsResults = new ArrayList();
Connection conn = null;
try
{
conn = this.dataSource.getConnection();
ResultSet rs = conn.createStatement().executeQuery(sql);
ResultSetMetaData data = rs.getMetaData();
int numberOfColumns = data.getColumnCount();
ListString lsRows = new ArrayList();
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(data.getColumnName(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
while (rs.next())
{
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(rs.getString(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
}
rs.close();
conn.close();
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
return (String[])lsResults.toArray(new String[lsResults.size()]);
}
On Mon, May 18, 2015 at 5:43 PM, Eli Levine elilev...@gmail.com wrote:
I don't have info on what your app does with results from Phoenix. If
the
app is constructing some sort of object representations from Phoenix
results
and holding on to them, I would look at what the memory footprint of
that
is. I know this isn't very helpful but at this point I would try to dig
deeper into your app and the NodeJS driver rather than Phoenix, since
you
mentioned the same queries run fine in sqlline.
On Mon, May 18, 2015 at 7:30 AM, Isart Montane isart.mont...@gmail.com
wrote:
Hi Eli,
thanks a lot for your answer. That might be a workaround but I was
hoping
to get a more generic answer I can apply to the driver/phoenix since
that
will require me lots of changes to the code.
Any clue on why it works with sqline but not trough the node driver?
On Mon, May 18, 2015 at 4:20 PM, Eli Levine elilev...@gmail.com
wrote:
Have you looked at paging [1] using Phoenix's row-value constructors
together with the LIMIT clause? That might be what you are looking
for.
[1] http://phoenix.apache.org/paged.html
Eli
On Mon, May 18, 2015 at 6:46 AM, Isart Montane
isart.mont...@gmail.com
wrote:
Hi,
the company I work for is performing some tests on Phoenix with
NodeJS.
For simple queries I didn't had any problem, but as soon as I start
to use
our app I'm getting process out of memory errors on the client
when I runs
queries that return a big number of rows (i.e. 400k) . I think the
problem
is that the client tries to buffer all the results in RAM and that
kills it.
The same query runs fine when I run it with sqline.
So, is there a way to tell the client to stream the results (or batch
them) instead of buffering them all? is raising the client memory
the only
solution?
I'm using phoenix-4.3.1 and
https://github.com/gaodazhu/phoenix-client
as the NodeJS driver
Thanks,
Isart Montane
Re: Phoenix and NodeJS
Yeah, so you can see that code creates a String array containing the whole
result set. Usually a very bad idea for 400K-row result sets. You want to
process results incrementally, probably via paging using row-value
constructors and LIMIT.
On Mon, May 18, 2015 at 12:00 PM, Isart Montane isart.mont...@gmail.com
wrote:
Thanks James.
That code is from the node driver, I will try to get some advice from it's
developer.
Thanks,
On Mon, May 18, 2015 at 6:34 PM, James Taylor jamestay...@apache.org
wrote:
Hi Isart,
That code isn't Phoenix code. This sounds like a Node JS issue. Vaclav
has done a lot with Node JS, so he may be able to give you some tips.
Thanks,
James
On Mon, May 18, 2015 at 9:06 AM, Isart Montane isart.mont...@gmail.com
wrote:
Hi Eli,
thanks a lot for your comments. I think you are right. I found the
client
code that's causing the issue. Do you have an example I can use to
patch it?
is that the recommended way to access phoenix? I've seen on the web that
there's also a query server available, is it worth a try?
public String[] query(String sql)
{
ListString lsResults = new ArrayList();
Connection conn = null;
try
{
conn = this.dataSource.getConnection();
ResultSet rs = conn.createStatement().executeQuery(sql);
ResultSetMetaData data = rs.getMetaData();
int numberOfColumns = data.getColumnCount();
ListString lsRows = new ArrayList();
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(data.getColumnName(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
while (rs.next())
{
for (int i = 1; i = numberOfColumns; i++) {
lsRows.add(rs.getString(i));
}
lsResults.add(join(\t, lsRows));
lsRows.clear();
}
rs.close();
conn.close();
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
return (String[])lsResults.toArray(new String[lsResults.size()]);
}
On Mon, May 18, 2015 at 5:43 PM, Eli Levine elilev...@gmail.com
wrote:
I don't have info on what your app does with results from Phoenix. If
the
app is constructing some sort of object representations from Phoenix
results
and holding on to them, I would look at what the memory footprint of
that
is. I know this isn't very helpful but at this point I would try to dig
deeper into your app and the NodeJS driver rather than Phoenix, since
you
mentioned the same queries run fine in sqlline.
On Mon, May 18, 2015 at 7:30 AM, Isart Montane
isart.mont...@gmail.com
wrote:
Hi Eli,
thanks a lot for your answer. That might be a workaround but I was
hoping
to get a more generic answer I can apply to the driver/phoenix since
that
will require me lots of changes to the code.
Any clue on why it works with sqline but not trough the node driver?
On Mon, May 18, 2015 at 4:20 PM, Eli Levine elilev...@gmail.com
wrote:
Have you looked at paging [1] using Phoenix's row-value constructors
together with the LIMIT clause? That might be what you are looking
for.
[1] http://phoenix.apache.org/paged.html
Eli
On Mon, May 18, 2015 at 6:46 AM, Isart Montane
isart.mont...@gmail.com
wrote:
Hi,
the company I work for is performing some tests on Phoenix with
NodeJS.
For simple queries I didn't had any problem, but as soon as I start
to use
our app I'm getting process out of memory errors on the client
when I runs
queries that return a big number of rows (i.e. 400k) . I think the
problem
is that the client tries to buffer all the results in RAM and that
kills it.
The same query runs fine when I run it with sqline.
So, is there a way to tell the client to stream the results (or
batch
them) instead of buffering them all? is raising the client memory
the only
solution?
I'm using phoenix-4.3.1 and
https://github.com/gaodazhu/phoenix-client
as the NodeJS driver
Thanks,
Isart Montane
Re: Can we create an secondary index for two or more columns?
Thank you Jesse, that really makes sense. 2015-05-18 11:19 GMT+08:00 Jesse Yates jesse.k.ya...@gmail.com: create index my_idx on EXAMPLE (M.C0, M.C1) This will create an index on both columns _at the same time_. This means the row key in the index will be a combination of both columns. Creating an index like this is only an advantage if you are commonly querying _both columns_ at the same time. For instance, a query like: Select * from EXAMPLE WHERE m.c0 = a AND m.c1 = b will leverage the index on both columns. However, if you are just querying each column separately, then using your solution (b) will be better. Does that make sense? --- Jesse Yates @jesse_yates jyates.github.com On Sun, May 17, 2015 at 6:17 PM, Tao Xiao xiaotao.cs@gmail.com wrote: Thanks to Yuhao, Is the column *M.C1* ignored when I invoke create index idx on EXAMPLE (M.C0, M.C1) ? Which of the following two solutions make sense? solution a): create index idx_1 on EXAMPLE (M.C0, M.C1) create index idx_2 on EXAMPLE (M.C1, M.C0) solution b): create index idx_1 on EXAMPLE (M.C0) create index idx_2 on EXAMPLE (M.C1) 2015-05-17 22:34 GMT+08:00 Yuhao Bi byh0...@gmail.com: Hi Xiao Tao, You can create another secondary index on the same columns. create index another_index on EXAMPLE(M.C1, M.C0) After doing that, query b) should return faster. 2015-05-17 22:25 GMT+08:00 Tao Xiao xiaotao.cs@gmail.com: I tried to build secondary index for two columns, *M.C0* and *M.C1*: create index my_idx on EXAMPLE (M.C0, M.C1) Then I tried to query by two indexes, respectively: a). select M.C0 from EXAMPLE where M.C0 = 'c0_0001' b). select M.C1 from EXAMPLE where M.C1 = 'c1_0001' Query a) returned results in less than 0.3 second, and query b) returned results in about 13 seconds. It seems that index was built for column *M.C0*, not *M.C1*. Can we build secondary index for two or more columns ? Thanks.
