Longer term we should look to this Bazel plugin to manage the Maven style build artifacts. We don't have to use it for publishing, but at least help us generate the jar and matching pom.xml with proper dependencies. I don't like having hard coded dependencies in a random pom.xml template. This tool seems like it could help with that. https://medium.com/wix-engineering/how-to-publish-artifacts-from-bazel-to-maven-central-71da0cf990f5
But this is work to be done after the RC. On Wed, Feb 23, 2022 at 7:08 PM Nicholas Nezis <[email protected]> wrote: > Well the issue raised was us having dependencies incorrectly shaded. So > this is one potential solution to that issue. We could decide to keep the > shading and just fix that if it is sufficient to do that. That might be > easier. But typically projects will provide the unshaded jar with > dependencies listed. In addition to Dave commenting on the weird shading, I > also had a team member of mine comment on the weirdness that Heron was > providing the shaded jar. But those were the two main motivators raising > the issue. > > On Wed, Feb 23, 2022 at 6:45 PM Josh Fischer <[email protected]> wrote: > >> Where did we decide we needed to change these poms? Was there an issue >> raised by one of the mentors? >> >> On Wed, Feb 23, 2022 at 4:06 PM Nicholas Nezis <[email protected]> >> wrote: >> >> > Yup. And how those pom.xml files relate to the generated jar files. One >> of >> > the issues was that we are shading dependencies into the jar. We should >> > instead be providing the jar with only our code, and the pom.xml should >> > include the dependencies. >> > >> > The scripts to generate the pom.xml files: >> > https://github.com/apache/incubator-heron/tree/master/scripts/ci >> > And the scripts to build the Heron API jar(s): >> > >> > >> https://github.com/apache/incubator-heron/blob/abb2767e3df3ca6eba009f46efe1f1e83695617a/heron/api/src/java/BUILD#L23-L87 >> > >> > "api-java-low-level": Normal "Topology API" jar >> > "api-java": Functional (i.e. Streamlet jar) >> > "api-java-low-level-functional": Combination of topology and streamlet >> code >> > "api-unshaded": Java Binary (why a binary???) with both, but kryo >> neverlink >> > dependency added. I think this might be for Storm compatibility? >> > "api-shaded": Remaps the protobuf classes based on this rule... but the >> > rule doesn't shade any of the other dependencies... If we will continue >> > shading, we should fix this. >> > >> > >> https://github.com/apache/incubator-heron/blob/master/heron/api/src/java/shade.conf >> > "heron-api": We create a copy of the heron-api.jar for some reason... no >> > idea why this is this way and we don't just rename the previouis build >> task >> > "classification": No idea what this is... is it part of the Heron API? >> Why >> > is it not included in the resulting "heron-api" jar? >> > >> > Questions I have: >> > 1. Do we want to keep shading dependencies, or should we include the >> > dependencies in the pom.xml? >> > 2. What is classification jar? >> > 3. We are publishing a Heron API jar (that final artifact), but our >> > examples are referencing the more immediate streamlet or low-level jars. >> > Should the examples reference that final jar? >> > >> > I've tried to be diligent in studying the code to make sure I don't >> break >> > anything. Perhaps Ning can suggest how to test I didn't break Storm >> > compatibility if I make changes. I don't know how to easily test that. >> Is >> > it covered in the CI testing? >> > >> > I've been super busy lately, so haven't gotten to make progress on this. >> > But I would love help if someone has the cycles. >> > >> > On Wed, Feb 23, 2022 at 11:22 AM Josh Fischer <[email protected]> >> wrote: >> > >> > > In relation to >> > > https://github.com/apache/incubator-heron/projects/7#card-76168234 >> > > >> > > Are these the poms you are talking about? >> > > >> > > https://github.com/apache/incubator-heron/tree/master/release/maven >> > > >> > > On Sat, Feb 12, 2022 at 3:24 PM Nicholas Nezis < >> [email protected] >> > > >> > > wrote: >> > > >> > > > I've been tracking the items here: >> > > > https://github.com/apache/incubator-heron/projects/7 >> > > > >> > > > I've been busy with work and life so haven't gotten to really do >> what I >> > > > wanted with the cleanup of the API jars. Not sure how best to test >> my >> > > > changes with Storm compatibility. I also started messing around >> with a >> > > new >> > > > Bazel plugin that generates the Pom files, but I think I'm going to >> > table >> > > > that for now. Will just update the existing pom.xml templates we >> have. >> > My >> > > > testing will be limited to submitting the samples to verify that >> they >> > > still >> > > > work. Any help testing my changes would be appreciated. I'll try to >> get >> > > > something that I can check in for review. >> > > > >> > > > On Sat, Feb 12, 2022 at 4:19 PM Josh Fischer <[email protected]> >> > > wrote: >> > > > >> > > > > Does anything else need to be done before we cut another rc?-- >> > > > > Sent from A Mobile Device >> > > > > >> > > > >> > > >> > >> >
