Thanks for the inputs. We shall go ahead and complete the release process with the existing 3.0.0-rc2 tag with no changes to bin.
Best regards, Janardhan On Fri, Jun 24, 2022, 01:01 arnab phani <phaniar...@gmail.com> wrote: > That is a good point. > The problem is relevant even for our binary release as we package the same > set of libraries for the binary and python packages. > For now, I suggest, we include all the packages from bin.xml. For the next > release, we need to do systematic experiments to remove unnecessary > libraries from bin/python. > > Regards, > Arnab.. > > On Thu, Jun 23, 2022 at 5:50 PM Baunsgaard, Sebastian > <baunsga...@tugraz.at.invalid> wrote: > > > Hi, > > > > To verify this we have our current "no environment test" in python this > > verify what packages are needed vs not. > > > > Unfortunately right now our main branch fails because of missing > packages. > > So i am making it bigger > > Ideally we would not need to pack any of the hadoop things into the > python > > package. > > Currently the system require hadoop jars because we import hadoop > packages > > many places in our code base where it could potentially be avoided. > > > > best regards > > Sebastian > > > > ________________________________ > > From: Janardhan <janard...@apache.org> > > Sent: Thursday, June 23, 2022 5:14:24 PM > > To: dev@systemds.apache.org > > Subject: Re: [DISCUSS] PyPi packages are more than 100 MB. > > > > Hi team, > > > > In the list attached before, the following > > > > 19MB - hadoop-client-api-3.3.1.jar [1] > > 31MB - hadoop-client-runtime-3.3.1.jar [2] > > > > are added, which are introduced in the Hadoop 3.x. > > > > These jars are added to the bin packaging, with > > `<include>*:hadoop-client*</include>`[3] line the bin.xml. It has not > > changed recently. > > > > Are these libraries intentional and important for binary release. Is > > it possible to remove them? > > > > > > [1] > https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client-api > > [2] > > > https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client-runtime > > [3] > > https://github.com/apache/systemds/blame/main/src/assembly/bin.xml#L100 > > [ > > > https://opengraph.githubassets.com/4346ffcbfafaa80de9f253ffae4064695cb243ef464a0b841cc5e00ee05f127b/apache/systemds > > ]< > https://github.com/apache/systemds/blame/main/src/assembly/bin.xml#L100> > > > > systemds/src/assembly/bin.xml at main · apache/systemds · GitHub< > > https://github.com/apache/systemds/blame/main/src/assembly/bin.xml#L100> > > github.com > > An open source ML system for the end-to-end data science lifecycle - > > systemds/src/assembly/bin.xml at main · apache/systemds > > > > > > > > Thanks, > > Janardhan > > > > > > > > On Tue, Jun 21, 2022 at 11:40 PM arnab phani <phaniar...@gmail.com> > wrote: > > > > > > I thought, we only include the libraries from SystemDS binary in the > > python > > > package. If so, then hadoop-* libraries are not new additions. > > > Unfortunately, test.pypi doesn't allow packages of more than 100MB, > which > > > means we won't be able to dry run our python releases. > > > I would be a little more comfortable with a better explanation for why > > the > > > python package size increased by 2x from the last release. > > > > > > Regards, > > > Arnab.. > > > > > > On Tue, Jun 21, 2022 at 6:55 PM Janardhan <janard...@apache.org> > wrote: > > > > > > > Hi, > > > > > > > > PyPi packages are a little more than 100MB. Compared 2.2.1 which is > > ~56 MB. > > > > > > > > -- Added in the present release (library sizes after unzip) > > > > > > > > 70K Jun 21 15:08 commons-compiler-3.0.16.jar > > > > 601K Jun 21 15:08 commons-compress-1.19.jar > > > > > > > > 193K Jun 21 15:08 commons-text-1.6.jar > > > > > > > > 19M Jun 21 15:08 hadoop-client-api-3.3.1.jar > > > > 31M Jun 21 15:08 hadoop-client-runtime-3.3.1.jar > > > > > > > > 5.3M Jun 21 15:08 hadoop-hdfs-client-3.3.1.jar > > > > > > > > 1.5M Jun 21 15:08 htrace-core4-4.1.0-incubating.jar > > > > > > > > 126K Jun 21 15:08 re2j-1.1.jar > > > > > > > > 192K Jun 21 15:08 stax2-api-4.2.1.jar > > > > 511K Jun 21 15:08 woodstox-core-5.3.0.jar > > > > > > > > Let us see if there is some optimization we can do? > > > > > > > > Best, > > > > Janardhan > > > > > > >
