Re-distribute Phoenix 4.8 or later without any License issues due to nested libraries

[Mallieswari Dineshbabu]

Hi Team,



We would like to include the following Phoenix *distribution* into our
product and re-distribute the same.

https://archive.apache.org/dist/phoenix/apache-phoenix-4.12.0-HBase-1.2/bin/



Though the License is listed as Apache License Version 2.0, there are *many
nested dependencies in the license file*.

https://github.com/apache/phoenix/blob/4.12-HBase-1.2/dev/release_files/LICENSE



Could you please clarify us the following,

   1. Can we *re-distribute* Phoenix distribution after doing some minor
   changes in conf files like sqlline.py to mention Hadoop path? (We will ship
   the License file as well)
   2. Will the nested libraries mentioned in License agreement will *not
   result with any problem* for us once we distribute?



Please let me know in case of any queries,




-- 
Thanks and regards
Mallieswari D

Re: SSL Phoenix

[Ash N]

Thanks Josh.

On Nov 28, 2017 11:24 AM, "Josh Elser"  wrote:

Have you read the portion of the HBase book that I previously linked to?
This is handled by SASL and GSSAPI/Kerberos. Please use your favorite
search engine and do some reading.

SSL is just *one* library that can be used to provide privacy of data in
motion.


On 11/27/17 7:25 AM, Ash N wrote:

> Josh,
>
> Thank you for your comment.
>
> 1.
> Could you please  point me to any resources around the below statement you
> make?
>
> " there are definitely the tools/configuration that exist to provide end
> to end data  privacy  "
>
> 2.SSL is just not part of that picture :)
>
> Above statement is contrary to my understanding.
>
> Thought SSL enables secure connections.
>
> Input as always is appropriated.
>
> Thanks.
>
>
> On Nov 26, 2017 8:58 PM, "Josh Elser" > wrote:
>
> Thanks, Ash. Just to confirm, there are definitely the
> tools/configuration that exist to provide end to end data privacy
> (at rest and in motion). SSL is just not part of that picture :)
>
> On Nov 24, 2017 12:19, "Ash N" <742...@gmail.com
> > wrote:
>
> Josh,
>
> Thank you for your quick response.
>
> The data is sensitive personal data of customers.  Everything
> needs to be encrypted and secure.  In - wire, on-wire,
> in-motion, at rest, everything.
> Our solution was to use SSL/TLS everywhere.  Our development
> team reported that Phoenix does not support SSL. Therefore this
> is a big problem.
>
> Based on the above statements,  if you have additional ideas, I
> will gladly take them,
> if you have additional input please do provide.  I unfortunately
> have very limited to no knowledge on security.  So this becomes
> a challenge area for me.
>
> Meanwhile,  I will look up the link you have provided and will
> continue to do research on this topic.
>
> thanks,
> -ash
>
> On Fri, Nov 24, 2017 at 12:11 PM, Josh Elser  > wrote:
>
> Why do you have a hard-requirement on using SSL?
>
> HBase itself does not use SSL to provide confidentiality on
> its wire communication, it relies on jGSS and SASL to
> implement this security. Under the hood, this actually boils
> down to using GSSAPI, Kerberos specifically, to implement
> privacy (e.g. aes256-cts-hmac-sha1-96).
>
> Take a look at
> https://hbase.apache.org/book.html#_server_side_configuratio
> n_for_secure_operation
>  on_for_secure_operation>.
>
> Phoenix executes all of its RPCs over HBase RPCs, so if you
> have HBase set up correctly, Phoenix will follow.
>
> If you want to introduce the Phoenix Query Server into your
> architecture, you can place it behind an SSL/TLS proxy
> server (or configure PQS directly with SSL/TLS using a
> sufficiently new version of Phoenix). This would be the only
> way I know of to "use Phoenix with SSL", but, in my
> experience, this is rarely what people actually want when
> they say this ;)
>
> Disclaimer: I have no idea how any of this translates to EMR :)
>
>
> On 11/24/17 12:01 PM, Ash N wrote:
>
> Hello All,
>
> Thank you for the great work the team is doing on Phoenix.
>
> Summary :  does Phoenix support SSL connection in Amazon
> EMR Cluster?
>
> We are running Phoenix on EMR cluster in Amazon. We have
> a need to connect to Phoenix over SSL.  I don't see much
> documentation around this topic anywhere also I saw a
> couple of jira tickets that did not provide enough help
> or direction on this topic.
>
> If Phoenix does not support SSL connections what are my
> options?
>
> Starting off six months ago,  we assumed this should not
> be an issue.  Now we are in big trouble.
>
> All and any help is greatly appreciated.
>
> Thanks
> Ash
>
>
>
>

Re: Bulk loading into table vs view

[Sergey Soldatov]

Please take a look at https://phoenix.apache.org/views.html
All views are 'virtual' tables, so they don't have a dedicated physical
table and operates on top of the table that is specified in the view DDL.

Thanks,
Sergey

On Sat, Nov 25, 2017 at 6:25 AM, Eisenhut, Roman 
wrote:

> Dear Phoenix-Team,
>
>
>
> I did some test on bulk-loading data with the psql.py script in
> $PHOENIX_HOME/bin and the tpc-h data on my cluster with 1 master and 3 RS.
> I’ve found that it makes quite a difference whether you:
>
>1. Create a table
>2. Bulk load data into that table
>
> Or
>
>1. Create a table
>2. Create a view
>3. Bulk load data in the view
>
>
>
> I was wondering where the overhead is coming from? (you can find my
> numbers below)
>
>
>
> Additionally, I created a view over a table which was already filled and
> phoenix returned “No rows affected”. At the same time I can’t find a table
> in HBase that reflects the view, which makes me wonder whether views are
> actually materialized somewhere. As I’m quite interested in the view
> functionality of Phoenix, I was wondering whether someone can explain what
> is happening when a view is created?
>
>
>
> Best regards,
>
> Roman
>
>
>
>
>
> *psql.py -t X -d '|' X.csv, where X = table name*
>
> *ID*
>
> *TABLE*
>
> *region*
>
> *nation*
>
> *supplier*
>
> *customer*
>
> *part*
>
> *partsupp*
>
> *orders*
>
> *lineitem*
>
> *5*
>
> *25.00*
>
> *10,000*
>
> *150,000*
>
> *200,000*
>
> *800,000*
>
> *1,500,000*
>
> *6,001,215*
>
> *1*
>
> 0.068
>
> 0.11
>
> 2.959
>
> 18.789
>
> 27.881
>
> 107.03
>
> 164.853
>
> 1007.315
>
> *2*
>
> 0.124
>
> 0.093
>
> 2.993
>
> 19.62
>
> 26.954
>
> 80.671
>
> 169.038
>
> 1039.294
>
> *3*
>
> 0.07
>
> 0.092
>
> 2.795
>
> 20.745
>
> 29.036
>
> 76.855
>
> 177.765
>
> 1042.642
>
> *4*
>
> 0.132
>
> 0.101
>
> 2.89
>
> 20.527
>
> 28.121
>
> 78.956
>
> 180.145
>
> 1019.047
>
> *5*
>
> 0.072
>
> 0.116
>
> 3.334
>
> 27.494
>
> 28.891
>
> 75.455
>
> 166.668
>
> 1011.299
>
> *MIN*
>
> 0.068
>
> 0.092
>
> 2.795
>
> 18.789
>
> 26.954
>
> 75.455
>
> 164.853
>
> 1007.315
>
> *MAX*
>
> 0.132
>
> 0.116
>
> 3.334
>
> 27.494
>
> 29.036
>
> 107.03
>
> 180.145
>
> 1042.642
>
> *AVG*
>
> 0.0932
>
> 0.1024
>
> 2.9942
>
> 21.435
>
> 28.1766
>
> 83.7934
>
> 171.6938
>
> 1023.919
>
>
>
> *psql.py -t X_VIEW -d '|' X.csv, where X = table name*
>
> *ID*
>
> *VIEW*
>
> *region*
>
> *nation*
>
> *supplier*
>
> *customer*
>
> *part*
>
> *partsupp*
>
> *orders*
>
> *lineitem*
>
> *5*
>
> *25.00*
>
> *10,000*
>
> *150,000*
>
> *200,000*
>
> *800,000*
>
> *1,500,000*
>
> *6,001,215*
>
> *1*
>
> 0.103
>
> 0.159
>
> 2.644
>
> 22.702
>
> 28.424
>
> 93.897
>
> 201.449
>
>
>
> *2*
>
> 0.097
>
> 0.138
>
> 2.641
>
> 20.926
>
> 32.014
>
> 95.195
>
> 190.939
>
>
>
> *3*
>
> 0.123
>
> 0.076
>
> 3.097
>
> 19.88
>
> 38.426
>
> 90.613
>
> 193.376
>
>
>
> *4*
>
> 0.092
>
> 0.098
>
> 3.14
>
> 23.522
>
> 29.509
>
> 99.443
>
> 192.348
>
>
>
> *5*
>
> 0.089
>
> 0.146
>
> 2.938
>
> 22.196
>
> 34.407
>
> 93.898
>
> 198.012
>
>
>
> *MIN*
>
> 0.089
>
> 0.076
>
> 2.641
>
> 19.88
>
> 28.424
>
> 90.613
>
> 190.939
>
> 0
>
> *MAX*
>
> 0.123
>
> 0.159
>
> 3.14
>
> 23.522
>
> 38.426
>
> 99.443
>
> 201.449
>
> 0
>
> *AVG*
>
> 0.1008
>
> 0.1234
>
> 2.892
>
> 21.8452
>
> 32.556
>
> 94.6092
>
> 195.2248
>
> #DIV/0!
>
>
>

Re: phoenix 4.13.0 on hbase 1.2

[James Taylor]

I've started a vote for Phoenix 4.13.1 which includes support again for
HBase 1.2. This is what the CDH based releases will be based off of
initially (i.e. the work Pedro has done on PHOENIX-4372).

Please check it out and vote on it. Thanks for volunteering to be release
manager going forward, Pedro.

 James

On Mon, Nov 13, 2017 at 10:17 AM, James Taylor 
wrote:

> We discussed whether or not we should continue with Phoenix releases for
> HBase 1.2, but no one showed any interested in being the release manager
> [1], so we concluded that we would stop doing them. It's important to
> remember that the ASF is a volunteer effort and anyone can step up and take
> on this responsibility. That's essentially how contributors build merit to
> become committers and eventually PMC members and the project continues to
> grow. If you're interested, I suggest you start a new DISCUSS thread on the
> dev list and volunteer. Here's what would need to be done:
> - cherry-pick changes from master between 4.12.0 and 4.13.0 release to
> 4.x-HBase-1.2 branch
> - create a pull request with the above and get a +1 from a committer
> - monitor the Jenkins job that'll run with these changes keeping a lookout
> for any test failures
> - assuming there are no test failures, follow the directions here [2] to
> perform a release
>
> Also, please make sure you subscribe to the dev list so you can
> participate in further discussions.
>
> Thanks,
> James
>
>
> [1] https://lists.apache.org/thread.html/ae13def3c024603ce3cdde8
> 71223cbdbae0219b4efe93ed4e48f55d5@%3Cdev.phoenix.apache.org%3E
> [2] https://phoenix.apache.org/release.html
>
> On Sun, Nov 12, 2017 at 11:38 PM, 최재환  wrote:
>
>> There is no compatibility with hbase 1.2 version in ANNOUNCE.
>>
>> i want to use phoenix 4.13.0 on hbase 1.2
>>
>> Are you planning to make hbase 1.2 compatible?
>>
>>
>>
>> *E-mail : prismso...@gmail.com *
>>
>
>

Re: 4.13.0-HBase-1.1 not released?

[James Taylor]

I've started a vote for Phoenix 4.13.1 which includes support again for
HBase 1.1. Please try it out and vote on the release. Thanks for
volunteering to be the release manager going forward, Xavier.

James

On Mon, Nov 20, 2017 at 8:09 AM, Xavier Jodoin  wrote:

> Hi James,
>
> Sorry for the delay I wasn't on the dev mailing list, I'm interested to
> help and I can take the lead for the Hbase 1.1 release.
>
> Xavier
> On 2017-11-18 03:22 PM, James Taylor wrote:
>
> FYI, we'll do one final release for Phoenix on HBase 1.1 (look for a
> 4.13.1 release soon). It looks like HBase 1.1 itself is nearing
> end-of-life, so probably good to move off of it. If someone is interested
> in being the RM for continued Phoenix HBase 1.1 releases, please volunteer.
>
> On Mon, Nov 13, 2017 at 10:24 AM, James R. Taylor 
> wrote:
>
>> Hi Xavier,
>> Please see these threads for some discussion. Would be great if you could
>> volunteer to be the release manager for Phoenix released on HBase 1.1.
>>
>> https://lists.apache.org/thread.html/8a73efa27edb70ea5cbc89b
>> 43c312faefaf2b78751c9459834523b81@%3Cuser.phoenix.apache.org%3E
>> https://lists.apache.org/thread.html/04de7c47724d8ef2ed7414d
>> 5bdc51325b2a0eecd324556d9e83f3718@%3Cdev.phoenix.apache.org%3E
>> https://lists.apache.org/thread.html/ae13def3c024603ce3cdde8
>> 71223cbdbae0219b4efe93ed4e48f55d5@%3Cdev.phoenix.apache.org%3E
>>
>> Thanks,
>> James
>>
>> On 2017-11-13 07:51, Xavier Jodoin  wrote:
>> > Hi,
>> >
>> > I would like to know if there is a reason why phoenix wasn't released
>> > for hbase 1.1?
>> >
>> > Thanks
>> >
>> > Xavier Jodoin
>> >
>> >
>>
>
>
>

Re: SSL Phoenix

[Josh Elser]

Have you read the portion of the HBase book that I previously linked to? 
This is handled by SASL and GSSAPI/Kerberos. Please use your favorite 
search engine and do some reading.


SSL is just *one* library that can be used to provide privacy of data in 
motion.


On 11/27/17 7:25 AM, Ash N wrote:

Josh,

Thank you for your comment.

1.
Could you please  point me to any resources around the below statement 
you make?


" there are definitely the tools/configuration that exist to provide end 
to end data  privacy  "


2.SSL is just not part of that picture :)

Above statement is contrary to my understanding.

Thought SSL enables secure connections.

Input as always is appropriated.

Thanks.


On Nov 26, 2017 8:58 PM, "Josh Elser" > wrote:


Thanks, Ash. Just to confirm, there are definitely the
tools/configuration that exist to provide end to end data privacy
(at rest and in motion). SSL is just not part of that picture :)

On Nov 24, 2017 12:19, "Ash N" <742...@gmail.com
> wrote:

Josh,

Thank you for your quick response.

The data is sensitive personal data of customers.  Everything
needs to be encrypted and secure.  In - wire, on-wire,
in-motion, at rest, everything.
Our solution was to use SSL/TLS everywhere.  Our development
team reported that Phoenix does not support SSL. Therefore this
is a big problem.

Based on the above statements,  if you have additional ideas, I
will gladly take them,
if you have additional input please do provide.  I unfortunately
have very limited to no knowledge on security.  So this becomes
a challenge area for me.

Meanwhile,  I will look up the link you have provided and will
continue to do research on this topic.

thanks,
-ash

On Fri, Nov 24, 2017 at 12:11 PM, Josh Elser > wrote:

Why do you have a hard-requirement on using SSL?

HBase itself does not use SSL to provide confidentiality on
its wire communication, it relies on jGSS and SASL to
implement this security. Under the hood, this actually boils
down to using GSSAPI, Kerberos specifically, to implement
privacy (e.g. aes256-cts-hmac-sha1-96).

Take a look at

https://hbase.apache.org/book.html#_server_side_configuration_for_secure_operation

.
Phoenix executes all of its RPCs over HBase RPCs, so if you
have HBase set up correctly, Phoenix will follow.

If you want to introduce the Phoenix Query Server into your
architecture, you can place it behind an SSL/TLS proxy
server (or configure PQS directly with SSL/TLS using a
sufficiently new version of Phoenix). This would be the only
way I know of to "use Phoenix with SSL", but, in my
experience, this is rarely what people actually want when
they say this ;)

Disclaimer: I have no idea how any of this translates to EMR :)


On 11/24/17 12:01 PM, Ash N wrote:

Hello All,

Thank you for the great work the team is doing on Phoenix.

Summary :  does Phoenix support SSL connection in Amazon
EMR Cluster?

We are running Phoenix on EMR cluster in Amazon. We have
a need to connect to Phoenix over SSL.  I don't see much
documentation around this topic anywhere also I saw a
couple of jira tickets that did not provide enough help
or direction on this topic.

If Phoenix does not support SSL connections what are my
options?

Starting off six months ago,  we assumed this should not
be an issue.  Now we are in big trouble.

All and any help is greatly appreciated.

Thanks
Ash