Re: [DISCUSS] Enabling CI for Oak cloud-based features

[Robert Munteanu]

On Tue, 2018-07-31 at 13:49 +0530, Chetan Mehrotra wrote:
> For Azure specific services we would need to
> get an account created and use that from our Jenkins server.

I know Tomek has been using azurite [1] with some success ( although
not 100% perfect ).

Robert

[1]: https://github.com/azure/azurite

Re: [DISCUSS] Enabling CI for Oak cloud-based features

[Chetan Mehrotra]

For S3 there are quite a few options and one suggested by Amit should
be good for our usecase. For Azure specific services we would need to
get an account created and use that from our Jenkins server.

Chetan Mehrotra

On Tue, Jul 31, 2018 at 1:45 PM Amit Jain  wrote:
>
> https://jclouds.apache.org/start/blobstore/
>
> On Tue, Jul 31, 2018 at 1:43 PM Amit Jain  wrote:
>
> > I don't have too much details but jclouds has some modules to persist to
> > filesystem and memory with the same API they have for other cloud backends.
> >
> >
> > On Tue, Jul 31, 2018 at 1:38 PM Michael Dürig  wrote:
> >
> >>
> >> I wonder how other communities address such issues. I.e. Apache jclouds
> >> would need to be tested against a rather broad variety of different
> >> cloud vendors. Maybe its worth to do some research on their approach.
> >>
> >> Michael
> >>
> >>
> >> On 31.07.18 10:01, Amit Jain wrote:
> >> > There's one provided by Adobe as well - https://github.com/adobe/S3Mock
> >> > But these would have to be enhanced to support the upload/download urls
> >> > which I don't think these support. Also, I am not aware of any similar
> >> > utility for Azure.
> >> >
> >> > Thanks
> >> > Amit
> >> >
> >> > On Tue, Jul 31, 2018 at 1:24 PM Julian Reschke 
> >> > wrote:
> >> >
> >> >> On 2018-07-30 19:26, Matt Ryan wrote:
> >> >>> Hi,
> >> >>>
> >> >>> Oak now has a fair few cloud-based modules - meaning, modules that
> >> enable
> >> >>> Oak to make use of cloud service provider capabilities in order for
> >> the
> >> >>> feature to work - among them being oak-blob-cloud,
> >> oak-blob-cloud-azure,
> >> >>> and oak-segment-azure.
> >> >>>
> >> >>> I’m not as familiar with oak-segment-azure, but I do know for
> >> >>> oak-blob-cloud and oak-blob-cloud-azure you need an environment set
> >> up to
> >> >>> run the tests including credentials for the corresponding cloud
> >> service
> >> >>> provider.  The consequence of this is that there is no regular CI
> >> testing
> >> >>> run on these modules, IIUC.
> >> >>>
> >> >>> I wanted to kick off a discussion to see what everyone else thinks.  I
> >> >>> think coming up with some form of mock for the cloud objects might be
> >> >> nice,
> >> >>> or even better to use existing Apache-license-friendly ones if there
> >> are
> >> >>> some, but maybe others have already gone down this road further or
> >> have
> >> >>> better ideas?
> >> >>> ...
> >> >>
> >> >> FWIW; this has been concerning me as well for quite some time.
> >> >>
> >> >> Maybe it's worth trying out ?
> >> >>
> >> >> Best regards, Julian
> >> >>
> >> >
> >>
> >

Re: [DISCUSS] Enabling CI for Oak cloud-based features

[Amit Jain]

https://jclouds.apache.org/start/blobstore/

On Tue, Jul 31, 2018 at 1:43 PM Amit Jain  wrote:

> I don't have too much details but jclouds has some modules to persist to
> filesystem and memory with the same API they have for other cloud backends.
>
>
> On Tue, Jul 31, 2018 at 1:38 PM Michael Dürig  wrote:
>
>>
>> I wonder how other communities address such issues. I.e. Apache jclouds
>> would need to be tested against a rather broad variety of different
>> cloud vendors. Maybe its worth to do some research on their approach.
>>
>> Michael
>>
>>
>> On 31.07.18 10:01, Amit Jain wrote:
>> > There's one provided by Adobe as well - https://github.com/adobe/S3Mock
>> > But these would have to be enhanced to support the upload/download urls
>> > which I don't think these support. Also, I am not aware of any similar
>> > utility for Azure.
>> >
>> > Thanks
>> > Amit
>> >
>> > On Tue, Jul 31, 2018 at 1:24 PM Julian Reschke 
>> > wrote:
>> >
>> >> On 2018-07-30 19:26, Matt Ryan wrote:
>> >>> Hi,
>> >>>
>> >>> Oak now has a fair few cloud-based modules - meaning, modules that
>> enable
>> >>> Oak to make use of cloud service provider capabilities in order for
>> the
>> >>> feature to work - among them being oak-blob-cloud,
>> oak-blob-cloud-azure,
>> >>> and oak-segment-azure.
>> >>>
>> >>> I’m not as familiar with oak-segment-azure, but I do know for
>> >>> oak-blob-cloud and oak-blob-cloud-azure you need an environment set
>> up to
>> >>> run the tests including credentials for the corresponding cloud
>> service
>> >>> provider.  The consequence of this is that there is no regular CI
>> testing
>> >>> run on these modules, IIUC.
>> >>>
>> >>> I wanted to kick off a discussion to see what everyone else thinks.  I
>> >>> think coming up with some form of mock for the cloud objects might be
>> >> nice,
>> >>> or even better to use existing Apache-license-friendly ones if there
>> are
>> >>> some, but maybe others have already gone down this road further or
>> have
>> >>> better ideas?
>> >>> ...
>> >>
>> >> FWIW; this has been concerning me as well for quite some time.
>> >>
>> >> Maybe it's worth trying out ?
>> >>
>> >> Best regards, Julian
>> >>
>> >
>>
>

Re: [DISCUSS] Enabling CI for Oak cloud-based features

[Amit Jain]

I don't have too much details but jclouds has some modules to persist to
filesystem and memory with the same API they have for other cloud backends.


On Tue, Jul 31, 2018 at 1:38 PM Michael Dürig  wrote:

>
> I wonder how other communities address such issues. I.e. Apache jclouds
> would need to be tested against a rather broad variety of different
> cloud vendors. Maybe its worth to do some research on their approach.
>
> Michael
>
>
> On 31.07.18 10:01, Amit Jain wrote:
> > There's one provided by Adobe as well - https://github.com/adobe/S3Mock
> > But these would have to be enhanced to support the upload/download urls
> > which I don't think these support. Also, I am not aware of any similar
> > utility for Azure.
> >
> > Thanks
> > Amit
> >
> > On Tue, Jul 31, 2018 at 1:24 PM Julian Reschke 
> > wrote:
> >
> >> On 2018-07-30 19:26, Matt Ryan wrote:
> >>> Hi,
> >>>
> >>> Oak now has a fair few cloud-based modules - meaning, modules that
> enable
> >>> Oak to make use of cloud service provider capabilities in order for the
> >>> feature to work - among them being oak-blob-cloud,
> oak-blob-cloud-azure,
> >>> and oak-segment-azure.
> >>>
> >>> I’m not as familiar with oak-segment-azure, but I do know for
> >>> oak-blob-cloud and oak-blob-cloud-azure you need an environment set up
> to
> >>> run the tests including credentials for the corresponding cloud service
> >>> provider.  The consequence of this is that there is no regular CI
> testing
> >>> run on these modules, IIUC.
> >>>
> >>> I wanted to kick off a discussion to see what everyone else thinks.  I
> >>> think coming up with some form of mock for the cloud objects might be
> >> nice,
> >>> or even better to use existing Apache-license-friendly ones if there
> are
> >>> some, but maybe others have already gone down this road further or have
> >>> better ideas?
> >>> ...
> >>
> >> FWIW; this has been concerning me as well for quite some time.
> >>
> >> Maybe it's worth trying out ?
> >>
> >> Best regards, Julian
> >>
> >
>

Re: [DISCUSS] Enabling CI for Oak cloud-based features

[Michael Dürig]

I wonder how other communities address such issues. I.e. Apache jclouds 
would need to be tested against a rather broad variety of different 
cloud vendors. Maybe its worth to do some research on their approach.


Michael


On 31.07.18 10:01, Amit Jain wrote:

There's one provided by Adobe as well - https://github.com/adobe/S3Mock
But these would have to be enhanced to support the upload/download urls
which I don't think these support. Also, I am not aware of any similar
utility for Azure.

Thanks
Amit

On Tue, Jul 31, 2018 at 1:24 PM Julian Reschke 
wrote:


On 2018-07-30 19:26, Matt Ryan wrote:

Hi,

Oak now has a fair few cloud-based modules - meaning, modules that enable
Oak to make use of cloud service provider capabilities in order for the
feature to work - among them being oak-blob-cloud, oak-blob-cloud-azure,
and oak-segment-azure.

I’m not as familiar with oak-segment-azure, but I do know for
oak-blob-cloud and oak-blob-cloud-azure you need an environment set up to
run the tests including credentials for the corresponding cloud service
provider.  The consequence of this is that there is no regular CI testing
run on these modules, IIUC.

I wanted to kick off a discussion to see what everyone else thinks.  I
think coming up with some form of mock for the cloud objects might be

nice,

or even better to use existing Apache-license-friendly ones if there are
some, but maybe others have already gone down this road further or have
better ideas?
...


FWIW; this has been concerning me as well for quite some time.

Maybe it's worth trying out ?

Best regards, Julian

Re: [DISCUSS] Enabling CI for Oak cloud-based features

[Amit Jain]

There's one provided by Adobe as well - https://github.com/adobe/S3Mock
But these would have to be enhanced to support the upload/download urls
which I don't think these support. Also, I am not aware of any similar
utility for Azure.

Thanks
Amit

On Tue, Jul 31, 2018 at 1:24 PM Julian Reschke 
wrote:

> On 2018-07-30 19:26, Matt Ryan wrote:
> > Hi,
> >
> > Oak now has a fair few cloud-based modules - meaning, modules that enable
> > Oak to make use of cloud service provider capabilities in order for the
> > feature to work - among them being oak-blob-cloud, oak-blob-cloud-azure,
> > and oak-segment-azure.
> >
> > I’m not as familiar with oak-segment-azure, but I do know for
> > oak-blob-cloud and oak-blob-cloud-azure you need an environment set up to
> > run the tests including credentials for the corresponding cloud service
> > provider.  The consequence of this is that there is no regular CI testing
> > run on these modules, IIUC.
> >
> > I wanted to kick off a discussion to see what everyone else thinks.  I
> > think coming up with some form of mock for the cloud objects might be
> nice,
> > or even better to use existing Apache-license-friendly ones if there are
> > some, but maybe others have already gone down this road further or have
> > better ideas?
> > ...
>
> FWIW; this has been concerning me as well for quite some time.
>
> Maybe it's worth trying out ?
>
> Best regards, Julian
>

Re: [DISCUSS] Enabling CI for Oak cloud-based features

[Julian Reschke]

On 2018-07-30 19:26, Matt Ryan wrote:

Hi,

Oak now has a fair few cloud-based modules - meaning, modules that enable
Oak to make use of cloud service provider capabilities in order for the
feature to work - among them being oak-blob-cloud, oak-blob-cloud-azure,
and oak-segment-azure.

I’m not as familiar with oak-segment-azure, but I do know for
oak-blob-cloud and oak-blob-cloud-azure you need an environment set up to
run the tests including credentials for the corresponding cloud service
provider.  The consequence of this is that there is no regular CI testing
run on these modules, IIUC.

I wanted to kick off a discussion to see what everyone else thinks.  I
think coming up with some form of mock for the cloud objects might be nice,
or even better to use existing Apache-license-friendly ones if there are
some, but maybe others have already gone down this road further or have
better ideas?
...


FWIW; this has been concerning me as well for quite some time.

Maybe it's worth trying out ?

Best regards, Julian

[DISCUSS] Enabling CI for Oak cloud-based features

[Matt Ryan]

Hi,

Oak now has a fair few cloud-based modules - meaning, modules that enable
Oak to make use of cloud service provider capabilities in order for the
feature to work - among them being oak-blob-cloud, oak-blob-cloud-azure,
and oak-segment-azure.

I’m not as familiar with oak-segment-azure, but I do know for
oak-blob-cloud and oak-blob-cloud-azure you need an environment set up to
run the tests including credentials for the corresponding cloud service
provider.  The consequence of this is that there is no regular CI testing
run on these modules, IIUC.

I wanted to kick off a discussion to see what everyone else thinks.  I
think coming up with some form of mock for the cloud objects might be nice,
or even better to use existing Apache-license-friendly ones if there are
some, but maybe others have already gone down this road further or have
better ideas?


-MR