Re: Cloudera parcel update

2017-12-11 Thread Flavio Pompermaier 
Can simeone replay correctly to this topic please?
http://community.cloudera.com/t5/Cloudera-Labs/Apache-Phoenix-Support-for-CDH-5-8-x-5-9x-5-10-x-5-11-x/m-p/62687#M416?eid=1=1

On 9 Nov 2017 21:36, "Pedro Boado"  wrote:

> I'll open a Jira ticket and put together a pull request in the next few
> days for Phoenix 4.11 + cdh 5.11.2 parcel.
>
> Then I'll start working on 4.12
>
> Sorry, I couldn't work on this in the last two weeks.
>
>
>
> On 9 Nov 2017 15:51, "James Taylor"  wrote:
>
>> I agree with JMS and there is interest from the PMC, but no bandwidth to
>> do the work - we’d look toward others like you to do the work of putting
>> together an initial pull request, regular pulls to keep things in sync,
>> RMing releases, etc. These types of contributions would earn merit toward a
>> committership and eventual nomination for PMC (that’s how the ASF works).
>> The files would of course be hosted on Apache infrastructure (just like our
>> current releases and repos).
>>
>> On Thu, Nov 9, 2017 at 6:15 AM Jean-Marc Spaggiari <
>> jean-m...@spaggiari.org> wrote:
>>
>>> We "simply" need to have a place to host the file, right? From a code
>>> perspective, it can be another branch in the same repo?
>>>
>>> 2017-11-09 8:48 GMT-05:00 Flavio Pompermaier :
>>>
 No interest from Phoenix PMCs to provide support to the creation of
 official Cloudera parcels (at least from Phoenix side)...?

 On Tue, Oct 31, 2017 at 8:09 AM, Flavio Pompermaier <
 pomperma...@okkam.it> wrote:

> Anyone from Phoenix...?
>
> On 27 Oct 2017 16:47, "Pedro Boado"  wrote:
>
>> For creating a CDH parcel repository the only thing needed is a web
>> server where the parcels and the manifest.json is published. But we need
>> one.
>>
>> I'm in of course. Who can help onboarding this changes and publishing
>> etc and getting users to push changes to the project? How do you do this 
>> in
>> Phoenix? Via another mail list, right?
>>
>> Defining regression strategy is probably the most complex bit. And
>> automating it is even more complex I think. This is where more work is
>> needed.
>>
>> On 27 Oct 2017 15:20, "Jean-Marc Spaggiari" 
>> wrote:
>>
>>> See below.
>>>
>>> 2017-10-27 8:45 GMT-04:00 Flavio Pompermaier :
>>>
 I just need someone who tells which git repository to use, the
 branching/tagging policy, what should be done to release a parcel (i.e.
 compile, test ok, update docs, etc). For example, I need someone who 
 says:
 to release a Phoenix CDH  parcel the process is this:

 1. use this repo (e.g.https://github.com/pboado
 /phoenix-for-cloudera
 
  or https://github.com/apahce/phoenix)

>>>
>>> Well, if Apache Phoenix maintains it, I feel this should be moved
>>> under the Apache Phoenix git repository, right?
>>>
>>>
>>>
 2. create one or more branches for each supported release (i.e.
 4.11-cdh-5.10 and 4.11-cdh-5.11)
 - this imply to create an official compatibility
 matrix...obviously it doesn't make sense to issue a 4.11-cdh-4.1 for
 example)

>>>
>>> Indeed.
>>>
>>>
 3. The test that should be passed to consider a parcel ok for a
 release

>>>
>>> Ha! good idea. Don't know if this can be automated, but deploying
>>> the parcel, doing rolling upgrades, minor versions and major versions
>>> upgrades tests, etc. We might be able to come with a list of things to
>>> test, and increase/improve the list as we move forward...
>>>
>>>
 4. Which documentation to write

>>>
>>> Most probably documenting what has changed between the Apache core
>>> branch and the updated parce branch? And how to build?
>>>
>>>
 5. Who is responsible to update Phoenix site and announcements etc?

>>>
>>> You? ;)
>>>
>>>
 6. Call for contributors when a new release is needed and
 coordinate them

>>>
>>> I'm already in! I have one CDH cluster and almost all CDH versions
>>> VMs... So I can do a lot of tests, as long as it doesn't required a 
>>> month
>>> of my time every month ;)
>>>
>>> JMS
>>>
>>>

 Kind of those things :)

 On Fri, Oct 27, 2017 at 2:33 PM, Jean-Marc Spaggiari <
 jean-m...@spaggiari.org> wrote:

> FYI, you can also count on me for that. At least to perform some
> testing or gather information, communication, etc.
>
> Flavio, what can you leading do you need there?
>
>

Re: Add automatic/default SALT

2017-12-11 Thread Josh Elser 
I'm a little hesitant of this for a few things I've noticed from lots of 
various installations:


* Salted tables are *not* always more efficient. In fact, I've found 
myself giving advice to not use salted tables a bit more than expected. 
Certain kinds of queries will require much more work if you have salting 
over not having salting


* Considering salt buckets as a measure of parallelism for a table, it's 
impossible for the system to correctly judge what the parallelism of the 
cluster should be. For example, with 10 RS and 1 Phoenix table, you 
would want to start with 10 salt buckets. However, with 10 RS and 100 
Phoenix tables, you'd *maybe* want to do 3 salt buckets. It's hard to 
make system wide decisions correctly without a global view of the entire 
system.


I think James was trying to capture some of this in his use of "relative 
conservative default", but I'd take that even a bit farther to say I 
consider it harmful for Phoenix to do that out of the box.


However, I would flip the question upside down instead: what kind of 
suggestions can Phoenix make as a database to the user to _recommend_ to 
them that they enable salting on a table given its schema and important 
queries?


On 12/8/17 12:34 PM, James Taylor wrote:

Hi Flavio,
I like the idea of “adaptable configuration” where you specify a config 
value as a % of some cluster resource (with relatively conservative 
defaults). Salting is somewhat of a gray area though as it’s not config 
based, but driven by your DDL. One solution you could implement on top 
of Phoenix is scripting for DDL that fills in the salt bucket parameter 
based on cluster size.

Thanks,
James

On Tue, Dec 5, 2017 at 12:50 AM Flavio Pompermaier > wrote:


Hi to all,
as stated by at the documentation[1] "for optimal performance,
number of salt buckets should match number of region servers".
So, why not to add an option AUTO/DEFAULT for salting that defaults
this parameter to the number of region servers?
Otherwise I have to manually connect to HBase, retrieve that number
and pass to Phoenix...
What do you think?

[1] https://phoenix.apache.org/performance.html#Salting

Best,
Flavio

Re: Problem With QueryServer In SQL Line Thin Client.

2017-12-11 Thread Josh Elser 

Weird, but glad you got it figured out.

PQS pulls the (thick) PhoenixDriver from the phoenix-client.jar you have 
in your Phoenix installation. Perhaps you had accidentally moved the jar 
aside and restarted PQS leaving PQS running without that jar? Hard to 
say, but that's my only guess.


On 12/8/17 7:18 AM, Vaghawan Ojha wrote:

Hi Josh,

I fixed it by specifying the host where the query server was running 
while doing python queryserver.py stop.


Something like this: python queryserver.py stop localhost:8765

And again started normally, it started working.

Thanks a lot.
Vaghawan

On Fri, Dec 8, 2017 at 7:38 AM, Vaghawan Ojha > wrote:


Hi Josh,

Here's the full stacktrace from PQS, I think its saying the PQS at
localhost has an error.

python sqlline-thin.py bigdata-namenode:8765
Failed to extract serialization from hbase-site.xml, defaulting to
PROTOBUF.
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect
jdbc:phoenix:thin:url=http://bigdata-namenode:8765;serialization=PROTOBUF
none none org.apache.phoenix.queryserver.client.Driver
Connecting to
jdbc:phoenix:thin:url=http://bigdata-namenode:8765;serialization=PROTOBUF
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in

[jar:file:/home/saque/hadoopec/phoenix/apache-phoenix-4.12.0-HBase-1.2-bin/phoenix-4.12.0-HBase-1.2-thin-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in

[jar:file:/home/saque/hadoopec/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
 for an explanation.
AvaticaClientRuntimeException: Remote driver error:
RuntimeException: java.sql.SQLException: No suitable driver found
for localhost -> SQLException: No suitable driver found for
localhost. Error -1 (0) null

java.lang.RuntimeException: java.sql.SQLException: No suitable
driver found for localhost
at
org.apache.calcite.avatica.jdbc.JdbcMeta.openConnection(JdbcMeta.java:621)
at
org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:285)
at

org.apache.calcite.avatica.remote.Service$OpenConnectionRequest.accept(Service.java:1771)
at

org.apache.calcite.avatica.remote.Service$OpenConnectionRequest.accept(Service.java:1751)
at

org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:94)
at

org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
at

org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127)
at org.apache.phoenix.shaded.org

.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.apache.phoenix.shaded.org

.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.apache.phoenix.shaded.org

.eclipse.jetty.server.Server.handle(Server.java:499)
at org.apache.phoenix.shaded.org

.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.apache.phoenix.shaded.org

.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.apache.phoenix.shaded.org
.eclipse.jetty.io

.AbstractConnection$2.run(AbstractConnection.java:544)
at org.apache.phoenix.shaded.org

.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.apache.phoenix.shaded.org

.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for localhost
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
org.apache.calcite.avatica.jdbc.JdbcMeta.openConnection(JdbcMeta.java:618)
... 15 more


at

org.apache.calcite.avatica.remote.Service$ErrorResponse.toException(Service.java:2476)
at

org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:63)
at

org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:81)
at org.apache.calcite.avatica.remote.Driver.connect(Driver.java:176)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
at