Re: Flink 1.10 on MapR secure cluster with high availability

2020-02-11 Thread Arvid Heise 
Hi Maxim,

in general, we have used shading in the past to avoid dependency hells for
users. We are in the process to replace more and more relocations with
plugins having a different classloader. However, I can't tell you if that
will also work for zookeeper and when that would happen.

If you manage to find out how Hadoop is solving that, I'd appreciate if you
could let us know.

Best,

Arvid

On Fri, Feb 7, 2020 at 5:02 PM Maxim Parkachov 
wrote:

> Hi Chesnay,
>
> I managed to re-compile with MapR zookeeper and can confirm that it works
> with HA as well.
> Still I find it strange that HA uses shadow version of zookeeper instead
> of version from classpath how it is done for hadoop.
>
> Thanks,
> Maxim.
>
> On Wed, Feb 5, 2020 at 3:43 PM Chesnay Schepler 
> wrote:
>
>> No, since a) HA will never use classes from the user-jar and b) zookeeper
>> is relocated to a different package (to avoid conflicts) and hence any
>> replacement has to follow the same relocation convention.
>>
>> On 05/02/2020 15:38, Maxim Parkachov wrote:
>>
>> Hi Chesnay,
>>
>> thanks for advise. Will it work if I include MapR specific zookeeper in
>> job dependencies and still use out-of-box Flink binary distribution ?
>>
>> Regards,
>> Maxim.
>>
>> On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler 
>> wrote:
>>
>>> You must rebuild Flink while overriding zookeeper.version property to
>>> match your MapR setup.
>>> For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604
>>> Note that you will also have to configure the MapR repository in your
>>> local setup as described here
>>> .
>>>
>>> On 05/02/2020 15:12, Maxim Parkachov wrote:
>>>
>>> Hi everyone,
>>>
>>> I have already written about issue with Flink 1.9 on secure MapR cluster
>>> and high availability. The issue was resolved with custom compiled Flink
>>> with vendor mapr repositories enabled. The history could be found
>>> https://www.mail-archive.com/user@flink.apache.org/msg28235.html
>>>
>>> Unfortunately, in current 1.10 RC vendor repositories were removed and
>>> I'm failing to get working configuration.  Current situation with 1.10 RC
>>> and secure MapR cluster:
>>>
>>> 1. Without HA, Flink uses class path provided zookeeper jar (mapr
>>> specific) and everything works fine.
>>>
>>> 2. With HA enabled, Flink uses shaded zookeeper  
>>> (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn)
>>> which  doesn't have MapR specific changes and fails to authenticate.
>>>
>>> I would really appreciate any help in resolving this issue.I'm ready to
>>> provide any required details.
>>>
>>> Regards,
>>> Maxim.
>>>
>>>
>>>
>>

Re: Flink 1.10 on MapR secure cluster with high availability

2020-02-07 Thread Maxim Parkachov 
Hi Chesnay,

I managed to re-compile with MapR zookeeper and can confirm that it works
with HA as well.
Still I find it strange that HA uses shadow version of zookeeper instead of
version from classpath how it is done for hadoop.

Thanks,
Maxim.

On Wed, Feb 5, 2020 at 3:43 PM Chesnay Schepler  wrote:

> No, since a) HA will never use classes from the user-jar and b) zookeeper
> is relocated to a different package (to avoid conflicts) and hence any
> replacement has to follow the same relocation convention.
>
> On 05/02/2020 15:38, Maxim Parkachov wrote:
>
> Hi Chesnay,
>
> thanks for advise. Will it work if I include MapR specific zookeeper in
> job dependencies and still use out-of-box Flink binary distribution ?
>
> Regards,
> Maxim.
>
> On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler 
> wrote:
>
>> You must rebuild Flink while overriding zookeeper.version property to
>> match your MapR setup.
>> For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604
>> Note that you will also have to configure the MapR repository in your
>> local setup as described here
>> .
>>
>> On 05/02/2020 15:12, Maxim Parkachov wrote:
>>
>> Hi everyone,
>>
>> I have already written about issue with Flink 1.9 on secure MapR cluster
>> and high availability. The issue was resolved with custom compiled Flink
>> with vendor mapr repositories enabled. The history could be found
>> https://www.mail-archive.com/user@flink.apache.org/msg28235.html
>>
>> Unfortunately, in current 1.10 RC vendor repositories were removed and
>> I'm failing to get working configuration.  Current situation with 1.10 RC
>> and secure MapR cluster:
>>
>> 1. Without HA, Flink uses class path provided zookeeper jar (mapr
>> specific) and everything works fine.
>>
>> 2. With HA enabled, Flink uses shaded zookeeper  
>> (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn)
>> which  doesn't have MapR specific changes and fails to authenticate.
>>
>> I would really appreciate any help in resolving this issue.I'm ready to
>> provide any required details.
>>
>> Regards,
>> Maxim.
>>
>>
>>
>

Re: Flink 1.10 on MapR secure cluster with high availability

2020-02-05 Thread Chesnay Schepler 
No, since a) HA will never use classes from the user-jar and b) 
zookeeper is relocated to a different package (to avoid conflicts) and 
hence any replacement has to follow the same relocation convention.


On 05/02/2020 15:38, Maxim Parkachov wrote:

Hi Chesnay,

thanks for advise. Will it work if I include MapR specific zookeeper 
in job dependencies and still use out-of-box Flink binary distribution ?


Regards,
Maxim.

On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler > wrote:


You must rebuild Flink while overriding zookeeper.version property
to match your MapR setup.
For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604
Note that you will also have to configure the MapR repository in
your local setup as described here
.

On 05/02/2020 15:12, Maxim Parkachov wrote:

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR
cluster and high availability. The issue was resolved with custom
compiled Flink with vendor mapr repositories enabled. The history
could be found
https://www.mail-archive.com/user@flink.apache.org/msg28235.html

Unfortunately, in current 1.10 RC vendor repositories were
removed and I'm failing to get working configuration.  Current
situation with 1.10 RC and secure MapR cluster:

1. Without HA, Flink uses class path provided zookeeper jar (mapr
specific) and everything works fine.

2. With HA enabled, Flink uses shaded zookeeper 
(org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn)
which doesn't have MapR specific changes and fails to authenticate.

I would really appreciate any help in resolving this issue.I'm
ready to provide any required details.

Regards,
Maxim.

Re: Flink 1.10 on MapR secure cluster with high availability

2020-02-05 Thread Maxim Parkachov 
Hi Chesnay,

thanks for advise. Will it work if I include MapR specific zookeeper in job
dependencies and still use out-of-box Flink binary distribution ?

Regards,
Maxim.

On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler  wrote:

> You must rebuild Flink while overriding zookeeper.version property to
> match your MapR setup.
> For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604
> Note that you will also have to configure the MapR repository in your
> local setup as described here
> .
>
> On 05/02/2020 15:12, Maxim Parkachov wrote:
>
> Hi everyone,
>
> I have already written about issue with Flink 1.9 on secure MapR cluster
> and high availability. The issue was resolved with custom compiled Flink
> with vendor mapr repositories enabled. The history could be found
> https://www.mail-archive.com/user@flink.apache.org/msg28235.html
>
> Unfortunately, in current 1.10 RC vendor repositories were removed and I'm
> failing to get working configuration.  Current situation with 1.10 RC and
> secure MapR cluster:
>
> 1. Without HA, Flink uses class path provided zookeeper jar (mapr
> specific) and everything works fine.
>
> 2. With HA enabled, Flink uses shaded zookeeper  
> (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn)
> which  doesn't have MapR specific changes and fails to authenticate.
>
> I would really appreciate any help in resolving this issue.I'm ready to
> provide any required details.
>
> Regards,
> Maxim.
>
>
>

Re: Flink 1.10 on MapR secure cluster with high availability

2020-02-05 Thread Chesnay Schepler 
You must rebuild Flink while overriding zookeeper.version property to 
match your MapR setup.

For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604
Note that you will also have to configure the MapR repository in your 
local setup as described here 
.


On 05/02/2020 15:12, Maxim Parkachov wrote:

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR 
cluster and high availability. The issue was resolved with custom 
compiled Flink with vendor mapr repositories enabled. The history 
could be found 
https://www.mail-archive.com/user@flink.apache.org/msg28235.html


Unfortunately, in current 1.10 RC vendor repositories were removed and 
I'm failing to get working configuration.  Current situation with 1.10 
RC and secure MapR cluster:


1. Without HA, Flink uses class path provided zookeeper jar (mapr 
specific) and everything works fine.


2. With HA enabled, Flink uses shaded zookeeper  
(org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn) 
which doesn't have MapR specific changes and fails to authenticate.


I would really appreciate any help in resolving this issue.I'm ready 
to provide any required details.


Regards,
Maxim.