Re: Cloudera parcel update

[Flavio Pompermaier]

Great! At this point we just need an official Phoenix mentor...

On Fri, Oct 27, 2017 at 4:19 PM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:

> See below.
>
> 2017-10-27 8:45 GMT-04:00 Flavio Pompermaier :
>
>> I just need someone who tells which git repository to use, the
>> branching/tagging policy, what should be done to release a parcel (i.e.
>> compile, test ok, update docs, etc). For example, I need someone who says:
>> to release a Phoenix CDH  parcel the process is this:
>>
>> 1. use this repo (e.g.https://github.com/pboado/phoenix-for-cloudera
>>  or
>> https://github.com/apahce/phoenix)
>>
>
> Well, if Apache Phoenix maintains it, I feel this should be moved under
> the Apache Phoenix git repository, right?
>
>
>
>> 2. create one or more branches for each supported release (i.e.
>> 4.11-cdh-5.10 and 4.11-cdh-5.11)
>> - this imply to create an official compatibility matrix...obviously
>> it doesn't make sense to issue a 4.11-cdh-4.1 for example)
>>
>
> Indeed.
>
>
>> 3. The test that should be passed to consider a parcel ok for a release
>>
>
> Ha! good idea. Don't know if this can be automated, but deploying the
> parcel, doing rolling upgrades, minor versions and major versions upgrades
> tests, etc. We might be able to come with a list of things to test, and
> increase/improve the list as we move forward...
>
>
>> 4. Which documentation to write
>>
>
> Most probably documenting what has changed between the Apache core branch
> and the updated parce branch? And how to build?
>
>
>> 5. Who is responsible to update Phoenix site and announcements etc?
>>
>
> You? ;)
>
>
>> 6. Call for contributors when a new release is needed and coordinate them
>>
>
> I'm already in! I have one CDH cluster and almost all CDH versions VMs...
> So I can do a lot of tests, as long as it doesn't required a month of my
> time every month ;)
>
> JMS
>
>
>>
>> Kind of those things :)
>>
>> On Fri, Oct 27, 2017 at 2:33 PM, Jean-Marc Spaggiari <
>> jean-m...@spaggiari.org> wrote:
>>
>>> FYI, you can also count on me for that. At least to perform some testing
>>> or gather information, communication, etc.
>>>
>>> Flavio, what can you leading do you need there?
>>>
>>> James, I am also interested ;) So count me in... (My very personal
>>> contribution)
>>>
>>> To setup a repo we just need to have a folder on the existing file
>>> storage with the correct parcel structure so people can point to it. That's
>>> not a big deal...
>>>
>>> JMS
>>>
>>> 2017-10-27 5:08 GMT-04:00 Flavio Pompermaier :
>>>
 I can give it a try..is there someone who can lead this thing?

>>>
>>>
>>
>>
>

Re: Cloudera parcel update

[Jean-Marc Spaggiari]

See below.

2017-10-27 8:45 GMT-04:00 Flavio Pompermaier :

> I just need someone who tells which git repository to use, the
> branching/tagging policy, what should be done to release a parcel (i.e.
> compile, test ok, update docs, etc). For example, I need someone who says:
> to release a Phoenix CDH  parcel the process is this:
>
> 1. use this repo (e.g.https://github.com/pboado/phoenix-for-cloudera
>  or
> https://github.com/apahce/phoenix)
>

Well, if Apache Phoenix maintains it, I feel this should be moved under the
Apache Phoenix git repository, right?



> 2. create one or more branches for each supported release (i.e.
> 4.11-cdh-5.10 and 4.11-cdh-5.11)
> - this imply to create an official compatibility matrix...obviously it
> doesn't make sense to issue a 4.11-cdh-4.1 for example)
>

Indeed.


> 3. The test that should be passed to consider a parcel ok for a release
>

Ha! good idea. Don't know if this can be automated, but deploying the
parcel, doing rolling upgrades, minor versions and major versions upgrades
tests, etc. We might be able to come with a list of things to test, and
increase/improve the list as we move forward...


> 4. Which documentation to write
>

Most probably documenting what has changed between the Apache core branch
and the updated parce branch? And how to build?


> 5. Who is responsible to update Phoenix site and announcements etc?
>

You? ;)


> 6. Call for contributors when a new release is needed and coordinate them
>

I'm already in! I have one CDH cluster and almost all CDH versions VMs...
So I can do a lot of tests, as long as it doesn't required a month of my
time every month ;)

JMS


>
> Kind of those things :)
>
> On Fri, Oct 27, 2017 at 2:33 PM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
>> FYI, you can also count on me for that. At least to perform some testing
>> or gather information, communication, etc.
>>
>> Flavio, what can you leading do you need there?
>>
>> James, I am also interested ;) So count me in... (My very personal
>> contribution)
>>
>> To setup a repo we just need to have a folder on the existing file
>> storage with the correct parcel structure so people can point to it. That's
>> not a big deal...
>>
>> JMS
>>
>> 2017-10-27 5:08 GMT-04:00 Flavio Pompermaier :
>>
>>> I can give it a try..is there someone who can lead this thing?
>>>
>>
>>
>
>

Re: Cloudera parcel update

[Jean-Marc Spaggiari]

FYI, you can also count on me for that. At least to perform some testing or
gather information, communication, etc.

Flavio, what can you leading do you need there?

James, I am also interested ;) So count me in... (My very personal
contribution)

To setup a repo we just need to have a folder on the existing file storage
with the correct parcel structure so people can point to it. That's not a
big deal...

JMS

2017-10-27 5:08 GMT-04:00 Flavio Pompermaier :

> I can give it a try..is there someone who can lead this thing?
>

Re: Cloudera parcel update

[Flavio Pompermaier]

I can give it a try..is there someone who can lead this thing?

Re: Cloudera parcel update

[James Taylor]

Yes, that's the idea I was floating - if you and Pedro (and others) are
interested, you could contribute the CDH compatible versions and these
could be bundled in our source and binary releases. The community,
including yourselves, would support it (in the same way we support Phoenix
today). Not sure what's involved in setting up a repo of Phoenix parcels,
but perhaps that can be done on Apache infrastructure too.


On Fri, Oct 27, 2017 at 1:22 AM, Flavio Pompermaier 
wrote:

> I totally agree wrt some compatibility matrix. I think that CDH
> compatibility and parcels release should be something that people of Apache
> Phoenix could support somehow..am I wrong?
>
> On Fri, Oct 27, 2017 at 10:06 AM, Pedro Boado 
> wrote:
>
>> There are problems regarding version support and bugfixes. As HBase in
>> CDH has been "enhaced" by Cloudera we'd need to mark certain specific cdh
>> versions as supported (up to major.minor.fix version) . For instance CDH
>> 5.11.1 doesn't pass tests because of an unsolved issue in Cloudera's HBase.
>>
>> I think this is not about sharing a parcel now and stop doing it, right?
>> Also there are "financial" issues in terms of servers that I'd probably
>> need help with.
>>
>> On 27 Oct 2017 08:49, "Flavio Pompermaier"  wrote:
>>
>>> If you need a hand I can help because we also need Phoenix on CDH. Maybe
>>> I could writie some documentation about it's installation and usage, on the
>>> README or on the official Phoenix site. Let's set up a an unofficial (but
>>> working) repo of Phoenix Parcels!
>>>
>>> On Fri, Oct 27, 2017 at 9:12 AM, Pedro Boado 
>>> wrote:
>>>
 Well I'd say it's the only option left to some of us to use Phoenix in
 our clients/companies. Most of them only go for commercial distributions of
 Hadoop for obvious reasons and cannot afford keeping up with frequent
 platform update. In our case each platform migration is bigger (in terms of
 effort etc) than some projects.

 Cloudera stopped porting it to CDH because of strategic reasons.
 Basically they say that Kudu does everything that Phoenix do... . Marketing
 strategy I guess. And we cannot stay with Phoenix 4.5 and CDH 5.5 (and no
 bugfixes) forever.

 In the other hand Hortonworks already keeps Phoenix within their own
 distribution so in this case the vendor itself sees strategic advantage on
 this.

 As long as I work for the same company I'll have to port Phoenix to
 CDH. So I could help with it guys.

 Maybe my only issue to publish the parcels is getting web hosting for
 them.



 On 27 Oct 2017 06:35, "James Taylor"  wrote:

> This is great, Pedro. Thanks so much for porting everything over.
> Would it make sense to try to have a CDH compatible release with each
> Phoenix release? Who would sign up to do this? Same question for HDP
> releases.
>
> Thanks,
> James
>
> On Thu, Oct 26, 2017 at 2:43 PM, Pedro Boado 
> wrote:
>
>> Sorry, it s provided "as is" . Try a "mvn clean package -D
>> skipTests=true" .
>>
>> And grab the parcel from phoenix-parcel/target
>>
>> On 26 Oct 2017 22:21, "Flavio Pompermaier" 
>> wrote:
>>
>>> And how do you use the parcel? Where it is generated?any
>>> documentation about this?
>>>
>>> On 26 Oct 2017 20:37, "Pedro Boado"  wrote:
>>>
 I've done it for Phoenix 4.11 and CDH 5.11.2 based on previous work
 from chiastic-security.

 https://github.com/pboado/phoenix-for-cloudera/tree/4.11-cdh5.11.2

 All integrations tests running, and I've added a parcel module for
 parcel-generation in rhel6.

 Contributions are welcome for supporting other parcel formats.

 I've also tried to compile Phoenix 4.12 but some IT are failing and
 I haven't looked further into them yet.

 On 26 Oct 2017 18:42, "Flavio Pompermaier" 
 wrote:

 I could give it a try...any reference about it?where can I find
 this latest parcel you produced? Any feedback from Cloudera?


 On 26 Oct 2017 18:38, "Jean-Marc Spaggiari" <
 jean-m...@spaggiari.org> wrote:

 It is. The parcel is not just a packaging of the Phoenix code into
 a different format. It requires some modifications. However, it's 
 doable...
 Andrew applied those modifications on a later version and we packaged 
 it
 into a Parcel. So it's definitely doable. Might be interesting to do 
 that
 for the last version, but will require some efforts...

 2017-10-26 12:07 GMT-04:00 Flavio Pompermaier 

Re: Cloudera parcel update

[Flavio Pompermaier]

I totally agree wrt some compatibility matrix. I think that CDH
compatibility and parcels release should be something that people of Apache
Phoenix could support somehow..am I wrong?

On Fri, Oct 27, 2017 at 10:06 AM, Pedro Boado  wrote:

> There are problems regarding version support and bugfixes. As HBase in CDH
> has been "enhaced" by Cloudera we'd need to mark certain specific cdh
> versions as supported (up to major.minor.fix version) . For instance CDH
> 5.11.1 doesn't pass tests because of an unsolved issue in Cloudera's HBase.
>
> I think this is not about sharing a parcel now and stop doing it, right?
> Also there are "financial" issues in terms of servers that I'd probably
> need help with.
>
> On 27 Oct 2017 08:49, "Flavio Pompermaier"  wrote:
>
>> If you need a hand I can help because we also need Phoenix on CDH. Maybe
>> I could writie some documentation about it's installation and usage, on the
>> README or on the official Phoenix site. Let's set up a an unofficial (but
>> working) repo of Phoenix Parcels!
>>
>> On Fri, Oct 27, 2017 at 9:12 AM, Pedro Boado 
>> wrote:
>>
>>> Well I'd say it's the only option left to some of us to use Phoenix in
>>> our clients/companies. Most of them only go for commercial distributions of
>>> Hadoop for obvious reasons and cannot afford keeping up with frequent
>>> platform update. In our case each platform migration is bigger (in terms of
>>> effort etc) than some projects.
>>>
>>> Cloudera stopped porting it to CDH because of strategic reasons.
>>> Basically they say that Kudu does everything that Phoenix do... . Marketing
>>> strategy I guess. And we cannot stay with Phoenix 4.5 and CDH 5.5 (and no
>>> bugfixes) forever.
>>>
>>> In the other hand Hortonworks already keeps Phoenix within their own
>>> distribution so in this case the vendor itself sees strategic advantage on
>>> this.
>>>
>>> As long as I work for the same company I'll have to port Phoenix to CDH.
>>> So I could help with it guys.
>>>
>>> Maybe my only issue to publish the parcels is getting web hosting for
>>> them.
>>>
>>>
>>>
>>> On 27 Oct 2017 06:35, "James Taylor"  wrote:
>>>
 This is great, Pedro. Thanks so much for porting everything over. Would
 it make sense to try to have a CDH compatible release with each Phoenix
 release? Who would sign up to do this? Same question for HDP releases.

 Thanks,
 James

 On Thu, Oct 26, 2017 at 2:43 PM, Pedro Boado 
 wrote:

> Sorry, it s provided "as is" . Try a "mvn clean package -D
> skipTests=true" .
>
> And grab the parcel from phoenix-parcel/target
>
> On 26 Oct 2017 22:21, "Flavio Pompermaier" 
> wrote:
>
>> And how do you use the parcel? Where it is generated?any
>> documentation about this?
>>
>> On 26 Oct 2017 20:37, "Pedro Boado"  wrote:
>>
>>> I've done it for Phoenix 4.11 and CDH 5.11.2 based on previous work
>>> from chiastic-security.
>>>
>>> https://github.com/pboado/phoenix-for-cloudera/tree/4.11-cdh5.11.2
>>>
>>> All integrations tests running, and I've added a parcel module for
>>> parcel-generation in rhel6.
>>>
>>> Contributions are welcome for supporting other parcel formats.
>>>
>>> I've also tried to compile Phoenix 4.12 but some IT are failing and
>>> I haven't looked further into them yet.
>>>
>>> On 26 Oct 2017 18:42, "Flavio Pompermaier" 
>>> wrote:
>>>
>>> I could give it a try...any reference about it?where can I find this
>>> latest parcel you produced? Any feedback from Cloudera?
>>>
>>>
>>> On 26 Oct 2017 18:38, "Jean-Marc Spaggiari" 
>>> wrote:
>>>
>>> It is. The parcel is not just a packaging of the Phoenix code into a
>>> different format. It requires some modifications. However, it's 
>>> doable...
>>> Andrew applied those modifications on a later version and we packaged it
>>> into a Parcel. So it's definitely doable. Might be interesting to do 
>>> that
>>> for the last version, but will require some efforts...
>>>
>>> 2017-10-26 12:07 GMT-04:00 Flavio Pompermaier 
>>> :
>>>
 Looking at [1] they say that "Attempt to run the official
 apache-phoenix-4.10.0-HBase-1.2 on CDH 5.12 failed due to tight
 coupling of CLAB_PHOENIX with HBase, and what looks like subtle API
 incompatibilities between Phoenix-embedded and CDH HBase classes.
 Namespace mapping partially worked, however". Is that true?

 [1] https://community.cloudera.com/t5/Cloudera-Labs/Apache-P
 hoenix-Support-for-CDH-5-8-x-5-9x-5-10-x-5-11-x/m-p/58729/hi
 ghlight/true#M378

 On Thu, Oct 26, 2017 at 12:13 AM, Flavio 

Re: Cloudera parcel update

[Pedro Boado]

There are problems regarding version support and bugfixes. As HBase in CDH
has been "enhaced" by Cloudera we'd need to mark certain specific cdh
versions as supported (up to major.minor.fix version) . For instance CDH
5.11.1 doesn't pass tests because of an unsolved issue in Cloudera's HBase.

I think this is not about sharing a parcel now and stop doing it, right?
Also there are "financial" issues in terms of servers that I'd probably
need help with.

On 27 Oct 2017 08:49, "Flavio Pompermaier"  wrote:

> If you need a hand I can help because we also need Phoenix on CDH. Maybe I
> could writie some documentation about it's installation and usage, on the
> README or on the official Phoenix site. Let's set up a an unofficial (but
> working) repo of Phoenix Parcels!
>
> On Fri, Oct 27, 2017 at 9:12 AM, Pedro Boado 
> wrote:
>
>> Well I'd say it's the only option left to some of us to use Phoenix in
>> our clients/companies. Most of them only go for commercial distributions of
>> Hadoop for obvious reasons and cannot afford keeping up with frequent
>> platform update. In our case each platform migration is bigger (in terms of
>> effort etc) than some projects.
>>
>> Cloudera stopped porting it to CDH because of strategic reasons.
>> Basically they say that Kudu does everything that Phoenix do... . Marketing
>> strategy I guess. And we cannot stay with Phoenix 4.5 and CDH 5.5 (and no
>> bugfixes) forever.
>>
>> In the other hand Hortonworks already keeps Phoenix within their own
>> distribution so in this case the vendor itself sees strategic advantage on
>> this.
>>
>> As long as I work for the same company I'll have to port Phoenix to CDH.
>> So I could help with it guys.
>>
>> Maybe my only issue to publish the parcels is getting web hosting for
>> them.
>>
>>
>>
>> On 27 Oct 2017 06:35, "James Taylor"  wrote:
>>
>>> This is great, Pedro. Thanks so much for porting everything over. Would
>>> it make sense to try to have a CDH compatible release with each Phoenix
>>> release? Who would sign up to do this? Same question for HDP releases.
>>>
>>> Thanks,
>>> James
>>>
>>> On Thu, Oct 26, 2017 at 2:43 PM, Pedro Boado 
>>> wrote:
>>>
 Sorry, it s provided "as is" . Try a "mvn clean package -D
 skipTests=true" .

 And grab the parcel from phoenix-parcel/target

 On 26 Oct 2017 22:21, "Flavio Pompermaier" 
 wrote:

> And how do you use the parcel? Where it is generated?any documentation
> about this?
>
> On 26 Oct 2017 20:37, "Pedro Boado"  wrote:
>
>> I've done it for Phoenix 4.11 and CDH 5.11.2 based on previous work
>> from chiastic-security.
>>
>> https://github.com/pboado/phoenix-for-cloudera/tree/4.11-cdh5.11.2
>>
>> All integrations tests running, and I've added a parcel module for
>> parcel-generation in rhel6.
>>
>> Contributions are welcome for supporting other parcel formats.
>>
>> I've also tried to compile Phoenix 4.12 but some IT are failing and I
>> haven't looked further into them yet.
>>
>> On 26 Oct 2017 18:42, "Flavio Pompermaier" 
>> wrote:
>>
>> I could give it a try...any reference about it?where can I find this
>> latest parcel you produced? Any feedback from Cloudera?
>>
>>
>> On 26 Oct 2017 18:38, "Jean-Marc Spaggiari" 
>> wrote:
>>
>> It is. The parcel is not just a packaging of the Phoenix code into a
>> different format. It requires some modifications. However, it's doable...
>> Andrew applied those modifications on a later version and we packaged it
>> into a Parcel. So it's definitely doable. Might be interesting to do that
>> for the last version, but will require some efforts...
>>
>> 2017-10-26 12:07 GMT-04:00 Flavio Pompermaier :
>>
>>> Looking at [1] they say that "Attempt to run the official
>>> apache-phoenix-4.10.0-HBase-1.2 on CDH 5.12 failed due to tight
>>> coupling of CLAB_PHOENIX with HBase, and what looks like subtle API
>>> incompatibilities between Phoenix-embedded and CDH HBase classes.
>>> Namespace mapping partially worked, however". Is that true?
>>>
>>> [1] https://community.cloudera.com/t5/Cloudera-Labs/Apache-P
>>> hoenix-Support-for-CDH-5-8-x-5-9x-5-10-x-5-11-x/m-p/58729/hi
>>> ghlight/true#M378
>>>
>>> On Thu, Oct 26, 2017 at 12:13 AM, Flavio Pompermaier <
>>> pomperma...@okkam.it> wrote:
>>>
 I'll take care of it ;)

 On 25 Oct 2017 23:45, "Sergey Soldatov" 
 wrote:

> Hi Flavio,
>
> It looks like you need to ask the vendor, not the community about
> their plan for further releases.
>
> Thanks,
> Sergey
>

Re: Cloudera parcel update

[Pedro Boado]

Well I'd say it's the only option left to some of us to use Phoenix in our
clients/companies. Most of them only go for commercial distributions of
Hadoop for obvious reasons and cannot afford keeping up with frequent
platform update. In our case each platform migration is bigger (in terms of
effort etc) than some projects.

Cloudera stopped porting it to CDH because of strategic reasons. Basically
they say that Kudu does everything that Phoenix do... . Marketing strategy
I guess. And we cannot stay with Phoenix 4.5 and CDH 5.5 (and no bugfixes)
forever.

In the other hand Hortonworks already keeps Phoenix within their own
distribution so in this case the vendor itself sees strategic advantage on
this.

As long as I work for the same company I'll have to port Phoenix to CDH. So
I could help with it guys.

Maybe my only issue to publish the parcels is getting web hosting for them.



On 27 Oct 2017 06:35, "James Taylor"  wrote:

> This is great, Pedro. Thanks so much for porting everything over. Would it
> make sense to try to have a CDH compatible release with each Phoenix
> release? Who would sign up to do this? Same question for HDP releases.
>
> Thanks,
> James
>
> On Thu, Oct 26, 2017 at 2:43 PM, Pedro Boado 
> wrote:
>
>> Sorry, it s provided "as is" . Try a "mvn clean package -D
>> skipTests=true" .
>>
>> And grab the parcel from phoenix-parcel/target
>>
>> On 26 Oct 2017 22:21, "Flavio Pompermaier"  wrote:
>>
>>> And how do you use the parcel? Where it is generated?any documentation
>>> about this?
>>>
>>> On 26 Oct 2017 20:37, "Pedro Boado"  wrote:
>>>
 I've done it for Phoenix 4.11 and CDH 5.11.2 based on previous work
 from chiastic-security.

 https://github.com/pboado/phoenix-for-cloudera/tree/4.11-cdh5.11.2

 All integrations tests running, and I've added a parcel module for
 parcel-generation in rhel6.

 Contributions are welcome for supporting other parcel formats.

 I've also tried to compile Phoenix 4.12 but some IT are failing and I
 haven't looked further into them yet.

 On 26 Oct 2017 18:42, "Flavio Pompermaier" 
 wrote:

 I could give it a try...any reference about it?where can I find this
 latest parcel you produced? Any feedback from Cloudera?


 On 26 Oct 2017 18:38, "Jean-Marc Spaggiari" 
 wrote:

 It is. The parcel is not just a packaging of the Phoenix code into a
 different format. It requires some modifications. However, it's doable...
 Andrew applied those modifications on a later version and we packaged it
 into a Parcel. So it's definitely doable. Might be interesting to do that
 for the last version, but will require some efforts...

 2017-10-26 12:07 GMT-04:00 Flavio Pompermaier :

> Looking at [1] they say that "Attempt to run the official
> apache-phoenix-4.10.0-HBase-1.2 on CDH 5.12 failed due to tight
> coupling of CLAB_PHOENIX with HBase, and what looks like subtle API
> incompatibilities between Phoenix-embedded and CDH HBase classes.
> Namespace mapping partially worked, however". Is that true?
>
> [1] https://community.cloudera.com/t5/Cloudera-Labs/Apache-P
> hoenix-Support-for-CDH-5-8-x-5-9x-5-10-x-5-11-x/m-p/58729/hi
> ghlight/true#M378
>
> On Thu, Oct 26, 2017 at 12:13 AM, Flavio Pompermaier <
> pomperma...@okkam.it> wrote:
>
>> I'll take care of it ;)
>>
>> On 25 Oct 2017 23:45, "Sergey Soldatov" 
>> wrote:
>>
>>> Hi Flavio,
>>>
>>> It looks like you need to ask the vendor, not the community about
>>> their plan for further releases.
>>>
>>> Thanks,
>>> Sergey
>>>
>>> On Wed, Oct 25, 2017 at 2:21 PM, Flavio Pompermaier <
>>> pomperma...@okkam.it> wrote:
>>>
 Hi to all,
 the latest Phoenix Cloudera parcel I can see is 4.7...any plan to
 release a newer version?

 I'd need at least Phoenix 4.9..anyone using it?

 Best,
 Flavio

>>>
>>>
>
>
>



>