Yup. And how those pom.xml files relate to the generated jar files. One of the issues was that we are shading dependencies into the jar. We should instead be providing the jar with only our code, and the pom.xml should include the dependencies.
The scripts to generate the pom.xml files: https://github.com/apache/incubator-heron/tree/master/scripts/ci And the scripts to build the Heron API jar(s): https://github.com/apache/incubator-heron/blob/abb2767e3df3ca6eba009f46efe1f1e83695617a/heron/api/src/java/BUILD#L23-L87 "api-java-low-level": Normal "Topology API" jar "api-java": Functional (i.e. Streamlet jar) "api-java-low-level-functional": Combination of topology and streamlet code "api-unshaded": Java Binary (why a binary???) with both, but kryo neverlink dependency added. I think this might be for Storm compatibility? "api-shaded": Remaps the protobuf classes based on this rule... but the rule doesn't shade any of the other dependencies... If we will continue shading, we should fix this. https://github.com/apache/incubator-heron/blob/master/heron/api/src/java/shade.conf "heron-api": We create a copy of the heron-api.jar for some reason... no idea why this is this way and we don't just rename the previouis build task "classification": No idea what this is... is it part of the Heron API? Why is it not included in the resulting "heron-api" jar? Questions I have: 1. Do we want to keep shading dependencies, or should we include the dependencies in the pom.xml? 2. What is classification jar? 3. We are publishing a Heron API jar (that final artifact), but our examples are referencing the more immediate streamlet or low-level jars. Should the examples reference that final jar? I've tried to be diligent in studying the code to make sure I don't break anything. Perhaps Ning can suggest how to test I didn't break Storm compatibility if I make changes. I don't know how to easily test that. Is it covered in the CI testing? I've been super busy lately, so haven't gotten to make progress on this. But I would love help if someone has the cycles. On Wed, Feb 23, 2022 at 11:22 AM Josh Fischer <[email protected]> wrote: > In relation to > https://github.com/apache/incubator-heron/projects/7#card-76168234 > > Are these the poms you are talking about? > > https://github.com/apache/incubator-heron/tree/master/release/maven > > On Sat, Feb 12, 2022 at 3:24 PM Nicholas Nezis <[email protected]> > wrote: > > > I've been tracking the items here: > > https://github.com/apache/incubator-heron/projects/7 > > > > I've been busy with work and life so haven't gotten to really do what I > > wanted with the cleanup of the API jars. Not sure how best to test my > > changes with Storm compatibility. I also started messing around with a > new > > Bazel plugin that generates the Pom files, but I think I'm going to table > > that for now. Will just update the existing pom.xml templates we have. My > > testing will be limited to submitting the samples to verify that they > still > > work. Any help testing my changes would be appreciated. I'll try to get > > something that I can check in for review. > > > > On Sat, Feb 12, 2022 at 4:19 PM Josh Fischer <[email protected]> > wrote: > > > > > Does anything else need to be done before we cut another rc?-- > > > Sent from A Mobile Device > > > > > >
