Hi! I have checked the OWASP output with Hadoop 3.3.6 and managed avro, woodstox and jettison versions. (It's too big to attach, but BUILDING.md includes the command line to generate it)
While about a third of them are false positives, there are a number of real ones. Opened https://issues.apache.org/jira/browse/PHOENIX-7405 for Jetty. We could bump jackson, though the CVE for it is disputed. htrace is also a problem, but I would not hold up 5.2.1 for removing it. For the others, we really need to think about where we draw the line. Let me know if there are other transitive dependencies we should meddle with. On Tue, Sep 10, 2024 at 1:54 PM Istvan Toth <st...@cloudera.com> wrote: > Thanks for the suggestion, but that is not realistic, HBase and Phoenix > are WAY too tightly coupled for that. > The Phoenix client JVM doesn't do that much, the majority of work and > interesting stuff happens in the HBase cluster. > > There is a subset of tests which CAN be run on separate clusters, and > there has been recent work done to improve on that. > Your suggestion of using Docker would make sense for those, for example it > could be used for testing interoperability of different Phoenix client and > server > versions, and upgrades (though we already have a - somewhat clumsy - test > framework for that) > > CLDR does have an internal test suite running on real clusters, 90% effort > and time is spent on configuring the clusters and re-starting them. > It is a good fit for some tests, mostly involving larger amounts of data, > and non-functional tests like using cloud connectors, but a bad fit for > functional / integration tests. > A similar docker-based system could probably improve on the configuration > complexity and restart times, but could still not be as fast as the > very stripped down minicluster. > > IUC SFDC is also working on something similar, but I don't know their > exact architecture for their internal tests. > > A significant amount of tests require very specifically configured HBase > instances with custom code and internal Hbase state management, which would > - take a huge amount of time and resources to start and run (as opposed to > the VERY optimized miniclusters) > - require bridging the different environments to manipulate and verify the > internal private state of the HBase cluster. > - require implementing a separate configuration management framework for > the stack. (something like Ambari) > > Phoenix already needs to be built for specific HBase versions, with each > branch only supporting a limited number of > HBase versions, for which we can run the tests (and we are already bad at > keeping the tests stable, even for our current coverage). > > On the other hand, Phoenix uses very few Hadoop features directly, it > mostly goes through HBase APIs. > All the heavy lifting for Hadoop compatibility is done in HBase, there is > very little to test from Phoenix itself in that regard. > > In short, this would take amount of resources - mostly in developers, but > also in computing resources for tests - that project does not have, for a > rather limited payoff. > > Istvan > > On Tue, Sep 10, 2024 at 11:17 AM Grzegorz Kokosiński < > g.kokosin...@gmail.com> wrote: > >> Hey, >> >> I am sorry I have joined the discussion late. >> >> If possible I would suggest to stop using mini clusters. I believe it is >> way more convenient to use docker with hadoop or hbase services. Notice >> that it is easy to test against multiple different versions, the >> environment is separated from the project classpath. In Trino that >> approach >> made testing easier. >> >> That way we could verify compatibility with old software even if we would >> be using newer libraries (hadoop and others). >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
