Ok, what I understand is that we need to fix this shading issue by hard coding dependencies into the pom for a short term win before creating the next RC. Long term we can look into this article https://medium.com/wix-engineering/how-to-publish-artifacts-from-bazel-to-maven-central-71da0cf990f5 for removing the hard coded deps in the pom file. If this is the case, no need to reply. If not, please let me know before I start to look into it.
On Wed, Feb 23, 2022 at 6:10 PM Nicholas Nezis <[email protected]> wrote: > Longer term we should look to this Bazel plugin to manage the Maven style > build artifacts. We don't have to use it for publishing, but at least help > us generate the jar and matching pom.xml with proper dependencies. I don't > like having hard coded dependencies in a random pom.xml template. This tool > seems like it could help with that. > > https://medium.com/wix-engineering/how-to-publish-artifacts-from-bazel-to-maven-central-71da0cf990f5 > > But this is work to be done after the RC. > > On Wed, Feb 23, 2022 at 7:08 PM Nicholas Nezis <[email protected]> > wrote: > > > Well the issue raised was us having dependencies incorrectly shaded. So > > this is one potential solution to that issue. We could decide to keep the > > shading and just fix that if it is sufficient to do that. That might be > > easier. But typically projects will provide the unshaded jar with > > dependencies listed. In addition to Dave commenting on the weird > shading, I > > also had a team member of mine comment on the weirdness that Heron was > > providing the shaded jar. But those were the two main motivators raising > > the issue. > > > > On Wed, Feb 23, 2022 at 6:45 PM Josh Fischer <[email protected]> > wrote: > > > >> Where did we decide we needed to change these poms? Was there an issue > >> raised by one of the mentors? > >> > >> On Wed, Feb 23, 2022 at 4:06 PM Nicholas Nezis < > [email protected]> > >> wrote: > >> > >> > Yup. And how those pom.xml files relate to the generated jar files. > One > >> of > >> > the issues was that we are shading dependencies into the jar. We > should > >> > instead be providing the jar with only our code, and the pom.xml > should > >> > include the dependencies. > >> > > >> > The scripts to generate the pom.xml files: > >> > https://github.com/apache/incubator-heron/tree/master/scripts/ci > >> > And the scripts to build the Heron API jar(s): > >> > > >> > > >> > https://github.com/apache/incubator-heron/blob/abb2767e3df3ca6eba009f46efe1f1e83695617a/heron/api/src/java/BUILD#L23-L87 > >> > > >> > "api-java-low-level": Normal "Topology API" jar > >> > "api-java": Functional (i.e. Streamlet jar) > >> > "api-java-low-level-functional": Combination of topology and streamlet > >> code > >> > "api-unshaded": Java Binary (why a binary???) with both, but kryo > >> neverlink > >> > dependency added. I think this might be for Storm compatibility? > >> > "api-shaded": Remaps the protobuf classes based on this rule... but > the > >> > rule doesn't shade any of the other dependencies... If we will > continue > >> > shading, we should fix this. > >> > > >> > > >> > https://github.com/apache/incubator-heron/blob/master/heron/api/src/java/shade.conf > >> > "heron-api": We create a copy of the heron-api.jar for some reason... > no > >> > idea why this is this way and we don't just rename the previouis build > >> task > >> > "classification": No idea what this is... is it part of the Heron API? > >> Why > >> > is it not included in the resulting "heron-api" jar? > >> > > >> > Questions I have: > >> > 1. Do we want to keep shading dependencies, or should we include the > >> > dependencies in the pom.xml? > >> > 2. What is classification jar? > >> > 3. We are publishing a Heron API jar (that final artifact), but our > >> > examples are referencing the more immediate streamlet or low-level > jars. > >> > Should the examples reference that final jar? > >> > > >> > I've tried to be diligent in studying the code to make sure I don't > >> break > >> > anything. Perhaps Ning can suggest how to test I didn't break Storm > >> > compatibility if I make changes. I don't know how to easily test that. > >> Is > >> > it covered in the CI testing? > >> > > >> > I've been super busy lately, so haven't gotten to make progress on > this. > >> > But I would love help if someone has the cycles. > >> > > >> > On Wed, Feb 23, 2022 at 11:22 AM Josh Fischer <[email protected]> > >> wrote: > >> > > >> > > In relation to > >> > > https://github.com/apache/incubator-heron/projects/7#card-76168234 > >> > > > >> > > Are these the poms you are talking about? > >> > > > >> > > https://github.com/apache/incubator-heron/tree/master/release/maven > >> > > > >> > > On Sat, Feb 12, 2022 at 3:24 PM Nicholas Nezis < > >> [email protected] > >> > > > >> > > wrote: > >> > > > >> > > > I've been tracking the items here: > >> > > > https://github.com/apache/incubator-heron/projects/7 > >> > > > > >> > > > I've been busy with work and life so haven't gotten to really do > >> what I > >> > > > wanted with the cleanup of the API jars. Not sure how best to test > >> my > >> > > > changes with Storm compatibility. I also started messing around > >> with a > >> > > new > >> > > > Bazel plugin that generates the Pom files, but I think I'm going > to > >> > table > >> > > > that for now. Will just update the existing pom.xml templates we > >> have. > >> > My > >> > > > testing will be limited to submitting the samples to verify that > >> they > >> > > still > >> > > > work. Any help testing my changes would be appreciated. I'll try > to > >> get > >> > > > something that I can check in for review. > >> > > > > >> > > > On Sat, Feb 12, 2022 at 4:19 PM Josh Fischer <[email protected] > > > >> > > wrote: > >> > > > > >> > > > > Does anything else need to be done before we cut another rc?-- > >> > > > > Sent from A Mobile Device > >> > > > > > >> > > > > >> > > > >> > > >> > > >
