Re: [VOTE]Release Apache Liminal (incubating) 0.0.5rc2-INCUBATING

[Aviem Zur]

+1 (non-binding)

On Mon, Jan 23, 2023 at 2:55 PM Lior Schachter  wrote:

> +1 (non binding)
>
> On Mon, 23 Jan 2023 at 12:16 Lidor Ettinger  wrote:
>
> > Hi All,
> >
> > We would like to vote on RC1 of version 0.0.5. The main purpose of V
> > 0.0.5 is to provide a clear extensibility API for developers to add
> > their own implementations of liminal abstractions - executor, task,
> > image builder. In addition this version supports Apache Airflow 2.0.
> >
> > Liminal community vote and result threads:
> > Vote:
> > https://lists.apache.org//thread/yftpyhfzszmhbtfr6xjdyhfkpyocfy11
> >
> > Result:
> > https://lists.apache.org//thread/9hsm5z4hygv4f0l059crp5bmp3hgg3m4
> >
> > Installation and testing instructions can be found in the README
> included.
> >
> > The release files, including signatures, digests, etc. can be found at:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/incubator/liminal/0.0.5rc2-INCUBATING/
> >
> >
> > The tag to be voted on:
> >
> >
> >
> https://github.com/apache/incubator-liminal/releases/tag/0.0.5rc2-INCUBATING
> >
> >
> > The SHA512 Checksum for these artifacts is:
> >
> > apache-liminal-0.0.5rc1-incubating-source.tar.gz:
> >
> >
> >
> 1937572ef1296d6f182ee855513b9884a290134d033afecaccb31c2208578d46395579192c37721d01e1807e3838915c96e55a8156e24871bf8aaf340e533500
> >
> >
> > Release artifacts are signed with the following key:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/liminal/KEYS
> >
> >
> > For more information about the contents of this release,
> > see:https://issues.apache.org/jira/projects/LIMINAL/versions/12352738
> >
> >
> > Please vote on releasing this package as Apache Liminal
> > 0.0.5-INCUBATING. A majority of at least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Liminal 0.0.5
> > [ ] +0 No opinion
> > [ ] -1 Do not release this package because ...
> >
> > Regards,
> > Lidor Ettinger
> >
>

Re: [VOTE] Release Apache Liminal (incubating) 0.0.2rc1-INCUBATING

[Aviem Zur]

Thanks for the update Ash.

On Mon, May 17, 2021 at 5:53 PM Ash Berlin-Taylor  wrote:

> Sorry to resurrect an old vote thread, but to follow up on this:
>
> In Airflow 2.1 we have removed _our_ cat-X core dep, but that doesn't help
> the Liminal project as they need the Docker and Kubernetes client libraries
> for their core.
>
> I have opened https://github.com/psf/requests/pull/5797 and Jarek has
> done a lot of the testing work, and it looks likely that the "requests"
> maintainers will accept this PR.
>
> And the other LGPL dep from the python Kubernetes client is in the process
> of changing to Apache2 license
> https://github.com/websocket-client/websocket-client/issues/665
>
> On 2021/04/21 14:55:14, Ash Berlin-Taylor  wrote:
> > Hi Aviem,
> >
> > Hi everyone, Airflow PMC here.
> >
> > First off, eek. Not sure how this happened, it's been like this for a
> while.
> >
> > I agree -- we could possibly fix the direct dependency on
> requests/chardet in Airflow, but as you have already discovered there are
> many more python libraries that depend upon requests - docker and
> kubernetes being the two that are hard to work around!.
> >
> > (Kubernetes has a _direct_ dep on a lgpl'd dep too)
> >
> > I'm writing a post to le...@apache.org, as _my_ reading is that
> depending upon LGPLd library at runtime only, but not shipping it is "work
> that uses the Library", and as per clause 5 of the LGPL 2.1 license should
> be fine https://opensource.org/licenses/LGPL-2.1
> >
> > > A program that contains no derivative of any portion of the Library,
> but is designed to work with the Library by being compiled or linked with
> it, is called a "work that uses the Library". Such a work, in isolation, is
> not a derivative work of the Library, and therefore falls outside the scope
> of this License.
> >
> > (There are a few caveats to this, so be careful about what else you
> might do with the code, such as creating)
> >
> > We'll see what Legal say about this.
> >
> > Thanks,
> > Ash
> >
> > On 2021/04/21 10:39:58, Aviem Zur  wrote:
> > > We can try to reach out. It looks like `chardet` can't change their
> license
> > > and `requests` have a won't fix / no time to fix attitude towards it.
> > >
> > > Meanwhile what do we do regarding releases of our artifacts?
> > > I don't see how we can develop a python project without having an
> indirect
> > > dependency on `requests` package as detailed above.
> > >
> > > On Wed, Apr 21, 2021 at 12:51 PM Justin Mclean <
> jus...@classsoftware.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Sorry I’m not sure what to suggest. Normally I’d say contact the
> project
> > > > and ask if they would be willing to distribute the library under
> another
> > > > license e.g. dual license it under LGPL and ALv2 but this seem
> unlikely
> > > > [1]. I guess it wouldn't hurt to ask again.
> > > >
> > > > Kind Regards,
> > > > Justin
> > > >
> > > >
> > > > 1. https://github.com/chardet/chardet/issues/36
> > > > -
> > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > > For additional commands, e-mail: general-h...@incubator.apache.org
> > > >
> > > >
> > >
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Liminal (incubating) 0.0.2rc1-INCUBATING

[Aviem Zur]

We can try to reach out. It looks like `chardet` can't change their license
and `requests` have a won't fix / no time to fix attitude towards it.

Meanwhile what do we do regarding releases of our artifacts?
I don't see how we can develop a python project without having an indirect
dependency on `requests` package as detailed above.

On Wed, Apr 21, 2021 at 12:51 PM Justin Mclean 
wrote:

> Hi,
>
> Sorry I’m not sure what to suggest. Normally I’d say contact the project
> and ask if they would be willing to distribute the library under another
> license e.g. dual license it under LGPL and ALv2 but this seem unlikely
> [1]. I guess it wouldn't hurt to ask again.
>
> Kind Regards,
> Justin
>
>
> 1. https://github.com/chardet/chardet/issues/36
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Liminal (incubating) 0.0.2rc1-INCUBATING

[Aviem Zur]

We declare exactly 9 dependencies in our requirements.txt


docker==4.2.0
apache-airflow==1.10.12
click==7.1.1
Flask==1.1.1
pyyaml==5.3.1
boto3==1.12.10
botocore==1.15.21
kubernetes==12.0.1
wheel==0.36.2


pipdeptree --reverse --packages chardet

chardet==3.0.4
  - requests==2.24.0 [requires: chardet>=3.0.2,<4]
- apache-airflow==1.10.12 [requires: requests>=2.20.0,<3]
- docker==4.2.0 [requires: requests>=2.14.2,!=2.18.0]
  - moto==1.3.14 [requires: docker>=2.5.1]
- kubernetes==12.0.1 [requires: requests]
- moto==1.3.14 [requires: requests>=2.5]
- requests-oauthlib==1.3.0 [requires: requests>=2.0.0]
  - kubernetes==12.0.1 [requires: requests-oauthlib]
- requests-toolbelt==0.9.1 [requires: requests>=2.0.1,<3.0.0]
  - twine==3.3.0 [requires: requests-toolbelt>=0.8.0,!=0.9.0]
- responses==0.12.1 [requires: requests>=2.0]
  - moto==1.3.14 [requires: responses>=0.9.0]
- Sphinx==3.5.1 [requires: requests>=2.5.0]
  - recommonmark==0.7.1 [requires: sphinx>=1.3.1]
  - sphinx-autobuild==2020.9.1 [requires: sphinx]
- twine==3.3.0 [requires: requests>=2.20]

As we can see above, many of the main packages we use require `requests`
which requires `chardet`

For example: airflow, docker, kubernetes, moto, sphinx all require
`requests` package which requires `chardet` package

Please advise,
Thanks,
Aviem


On Wed, Apr 21, 2021 at 8:22 AM Jean-Baptiste Onofre 
wrote:

> Hi Justin,
>
> I agree, with an updated DISCLAIMER, I guess we can move forward with
> 0.0.2 release.
>
> However, we have to double check that chardet is not a "direct" dependency
> (it seems to come from Airflow, but I didn’t actually check yet).
>
> Regards
> JB
>
> > Le 21 avr. 2021 à 00:15, Justin Mclean  a
> écrit :
> >
> > Hi,
> >
> >> And that’s my point: chardet is not a direct Liminal dependency, but
> actually an Airflow dependency.
> >>
> >> So, I think that’s either Liminal can release "as it is" or Airflow
> should address this dependency.
> >
> > If you had a WIP-DISCLAIMER and listed this issue in it you could make a
> release.
> >
> > I wold contact Airflow and have a chat to them and see why it needed.
> >
> > BTW I’m not 100% sure it is an Airflow dependancy, but I’m not 100%
> familiar with how python pip tools work.
> >
> > pipdeptree for me gives:
> >
> > pipreqs==0.4.10
> >  - docopt [required: Any, installed: 0.6.2]
> >  - yarg [required: Any, installed: 0.1.9]
> >- requests [required: Any, installed: 2.25.1]
> >  - certifi [required: >=2017.4.17, installed: 2020.4.5.1]
> >  - chardet [required: >=3.0.2,<5, installed: 4.0.0]
> >  - idna [required: >=2.5,<3, installed: 2.10]
> >  - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.2]
> >
> > pipdeptree --reverse --packages chardet gives:
> >
> > chardet==4.0.0
> >  - requests==2.25.1 [requires: chardet>=3.0.2,<5]
> >- yarg==0.1.9 [requires: requests]
> >  - pipreqs==0.4.10 [requires: yarg]
> >
> > Which is different to what you have.
> >
> > Pipregs is just for producing requirements.txt right? So it may not be a
> runtime dependancy and be considered optional?
> >
> > From  running  pip-licenses Yargs is MIT, requests is ALv2 and chardet
> is LGPLv2.
> >
> > Thanks,
> > Justin
> >
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Liminal (incubating) 0.0.1rc4-INCUBATING

[Aviem Zur]

We'll also compile a list of known issues and include it in disclaimer

Thanks,
Aviem

On Thu, Jan 28, 2021 at 10:14 AM Aviem Zur  wrote:

> Will add DISCLAIMER file to whl
>
> On Thu, Jan 28, 2021 at 8:42 AM Justin Mclean 
> wrote:
>
>> Hi,
>>
>> > ▶ shasum -c apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512
>> > shasum: apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512: No
>> such file or directory
>> >
>>⍉
>> > ▶ shasum -a 512  -c
>> apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512
>> > shasum: apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512: No
>> such file or directory
>>
>> This may help.
>>
>> The files may download with .txt extensions you may need to rename them
>> to .sha.
>>
>> I have a little bash function to do this:
>>
>> function untxt() {
>>  find . -name '*.txt' | while read f; do mv "$f" "${f/\.txt/}"; done
>> }
>>
>> I generally compare:
>> shasum -a 512 apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz
>>
>> with the .sha512 file under nuttx-ing it
>>
>> Thanks,
>> Justin
>> -
>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
>> For additional commands, e-mail: general-h...@incubator.apache.org
>>
>>

Re: [VOTE] Release Apache Liminal (incubating) 0.0.1rc4-INCUBATING

[Aviem Zur]

Will add DISCLAIMER file to whl

On Thu, Jan 28, 2021 at 8:42 AM Justin Mclean 
wrote:

> Hi,
>
> > ▶ shasum -c apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512
> > shasum: apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512: No such
> file or directory
> >
>  ⍉
> > ▶ shasum -a 512  -c
> apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512
> > shasum: apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz.sha512: No such
> file or directory
>
> This may help.
>
> The files may download with .txt extensions you may need to rename them to
> .sha.
>
> I have a little bash function to do this:
>
> function untxt() {
>  find . -name '*.txt' | while read f; do mv "$f" "${f/\.txt/}"; done
> }
>
> I generally compare:
> shasum -a 512 apache-liminal-0.0.1rc4-INCUBATING-source.tar.gz
>
> with the .sha512 file under nuttx-ing it
>
> Thanks,
> Justin
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Liminal (incubating) 0.0.1-INCUBATING

[Aviem Zur]

Hi,

It is a tar that is created as part of the python setup process.
We will not include it in releases going forward

Thanks,
aviem

On Fri, Jan 22, 2021 at 1:05 AM Justin Mclean 
wrote:

> Hi,
>
> > We will fix the issues and create a new release candidate.
>
> I’m still little confused why you have two release artefacts, if you can
> explain that it would be helpful.
>
> Thanks,
> Justin
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Accept Rainbow into Apache Incubator

[Aviem Zur]

Not sure what legally needs to be checked, but a quick search on uspto.gov
did not return any result of the single word 'rainbow':

http://tmsearch.uspto.gov/bin/showfield?f=toc=4805%3A82m03t.1.1_search=searchss_L=50=_plural=yes_s_PARA1=_tagrepl%7E%3A=PARA1%24LD=PARA1+AND+PARA2_s_PARA2=rainbow_tagrepl%7E%3A=PARA2%24COMB_op_ALL=AND_default=search_search=Submit+Query_search=Submit+Query

On Sun, May 3, 2020 at 2:00 PM Justin Mclean 
wrote:

> Hi,
>
> > We have conducted a name search for 'Rainbow'. We feel that although
> there
> > are few existing open source projects with this name, they are in a
> > different space, hence not necessarily restrict us from using it.
>
> Did you look at existing trademarks?
>
> Thanks,
> Justin
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Accept Rainbow into Apache Incubator

[Aviem Zur]

Hi,

Here is the link to the public repo:
https://github.com/Natural-Intelligence/rainbow
We are currently in the process of transferring our code to open-sourceable
version into this repo.

Thanks

On Mon, Mar 23, 2020 at 8:41 AM Jean-Baptiste Onofre 
wrote:

> Hi,
>
> Dave asked for the link to the private repo with the current codebase. I
> will check with the team to push the codebase on a public repo to share the
> current state.
>
> Anyway, any other vote/comment/request regarding Rainbow ?
>
> Thanks,
> Regards
> JB
>
> > Le 10 mars 2020 à 07:29, Jean-Baptiste Onofre  a écrit
> :
> >
> > Hi,
> >
> > Following the discussion about Rainbow proposal [1], I would like to
> call a vote to accept Rainbow [2] into the Apache Incubator.
> >
> > Please vote to accept Rainbow:
> >
> >  [ ] +1, accept Rainbow into Incubator
> >  [ ] -1, don’t accept Rainbow into Incubator (please provide a reason)
> >
> > The vote is open for 72 hours.
> >
> > Thanks,
> > Regards
> > JB
> >
> > [1]
> https://mail-archives.apache.org/mod_mbox/incubator-general/202002.mbox/%3CCADa_fpAzW9j283xv0ObUVgddNv-AqMC%2BDDmsW-8dmK-pzffMoA%40mail.gmail.com%3E
> <
> https://mail-archives.apache.org/mod_mbox/incubator-general/202002.mbox/%3ccada_fpazw9j283xv0obuvgddnv-aqmc+ddmsw-8dmk-pzff...@mail.gmail.com%3E
> >
> > [2] https://cwiki.apache.org/confluence/display/INCUBATOR/Apache+Rainbow
> 
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Project proposal - Apache Rainbow

[Aviem Zur]

Hi all,

Thanks for the feedback.

1. Indeed, the codebase is still under a private repository. We intend to
have it ready to share publicly later this March.
2. The project is built in Python and Java.This is due to the fact that we
have deep integrations with open source projects written in these languages.
We also considered the fact that it is used by both data scientists and
data engineers and we believe a combination of Python/Java will promote
collaboration and contribution.
3. Rainbow project intends to facilitate and simplify the composition of
complex pipelines, which are based on other open source projects.
As such it does not compete or overlap but rather complement these projects.
4. Re: DLAB project - as we see it this project focuses in the research
phase, while Rainbow's focus is in the production phase.
Seems the 2 projects complement each other and it would be very interesting
for us to collaborate with the DLAB team.
5. We will adjust the proposal to provide more details on how other Apache
projects are used in Rainbow.
We currently mainly use Apache Airflow in order to run pipelines defined by
users in our APIs (YAML, with plans of UI/REST), this reduces the
engineering requirements for transitioning data science code into
production. We also leverage Apache Spark and Apache Hive for data
preparation features and there are plans to integrate with Apache Karaf as
well.

Thanks,
Aviem

On Sat, Feb 22, 2020 at 4:29 AM Paul King  wrote:

> Indeed, it does sound interesting.
>
> I would find it useful if the "existing Apache projects" bit of "Rainbow is
> in development, leveraging existing Apache projects." could be expanded in
> any way. I know there is a list of external dependencies later but  any
> further description of how those technologies are used would be helpful.
>
> Also, I'd be interested in knowing how the proposal relates to DLAB:
> https://dlab.apache.org/
>
> Nice work.
>
> Cheers, Paul.
>
>
>
> On Sat, Feb 22, 2020 at 2:34 AM larry mccay  wrote:
>
> > This seems like an interesting proposal.
> >
> > Couple points/questions:
> >
> > * The existing source is not available for viewing as it is still in
> > private repos?
> > * Is it a primarily java project?
> > * It seems the intent of Rainbow is to not compete or overlap with the
> > Hadoop ecosystem projects but rather to provide an efficient interface
> > above them - correct?
> >
> >
> > On Fri, Feb 21, 2020 at 8:51 AM Aviem Zur  wrote:
> >
> > > Hi,
> > >
> > > We would like to propose Rainbow as an Apache incubator project.
> Rainbow
> > is
> > > an end-to-end platform for data engineers & scientists, allowing them
> to
> > > build, train and deploy machine learning models in a robust and agile
> > way.
> > > The project's goal is to operationalize the machine learning process,
> > > allowing data scientists to quickly transition from a successful
> > experiment
> > > to an automated pipeline in production.
> > >
> > > The proposal can be found here:
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/Apache+Rainbow
> > >
> > > We would appreciate your feedback and thoughts on the proposal.
> > >
> > > Thanks,
> > > Aviem
> > >
> >
>

[DISCUSS] Project proposal - Apache Rainbow

[Aviem Zur]

Hi,

We would like to propose Rainbow as an Apache incubator project. Rainbow is
an end-to-end platform for data engineers & scientists, allowing them to
build, train and deploy machine learning models in a robust and agile way.
The project's goal is to operationalize the machine learning process,
allowing data scientists to quickly transition from a successful experiment
to an automated pipeline in production.

The proposal can be found here:
https://cwiki.apache.org/confluence/display/INCUBATOR/Apache+Rainbow

We would appreciate your feedback and thoughts on the proposal.

Thanks,
Aviem