Julian, are you proposing the arrow project ship two artifacts, arrow-common and arrow, where arrow depends on arrow-common? On Mon, Feb 27, 2017 at 11:51 Julian Hyde <jh...@apache.org> wrote:
> “Commons” projects are often problematic. It is difficult to tell what is > in scope and out of scope. If the scope is drawn too wide, there is a real > problem of orphaned features, because people contribute one feature and > then disappear. > > Let’s remember the Apache mantra: community over code. If you create a > sustainable community, the code will get looked after. Would this project > form a new community, or just a new piece of code? As I read the current > proposal, it would be the intersection of some existing communities, not a > new community. > > I think it would take a considerable effort to create a new project and > community around the idea of “c++ commons” (or is it “database-related c++ > commons”?). I think you already have such a community, to a first > approximation, in the Arrow project, because Kudu and Impala developers are > already part of the Arrow community. There’s no reason why Arrow cannot > contain new modules that have different release schedules than the rest of > Arrow. As a TLP, releases are less burdensome, and can happen in a little > over 3 days if the component is kept stable. > > Lastly, the code is fungible. It can be marked “experimental” within Arrow > and moved to another project, or into a new project, as it matures. The > Apache license and the ASF CLA makes this very easy. We are doing something > like this in Calcite: the Avatica sub-project [1] has a community that > intersect’s with Calcite’s, is disconnected at a code level, and may over > time evolve into a separate project. In the mean time, being part of an > established project is helpful, because there are PMC members to vote. > > Julian > > [1] https://calcite.apache.org/avatica/ < > https://calcite.apache.org/avatica/> > > > On Feb 27, 2017, at 6:41 AM, Wes McKinney <wesmck...@gmail.com> wrote: > > > > Responding to Todd's e-mail: > > > > 1) Open source release model > > > > My expectation is that this library would release about once a month, > > with occasional faster releases for critical fixes. > > > > 2) Governance/review model > > > > Beyond having centralized code reviews, it's hard to predict how the > > governance would play out. I understand that OSS projects behave > > differently in their planning / design / review process, so work on a > > common need may require more of a negotiation than the prior > > "unilateral" process. > > > > I think it says something for our communities that we would make a > > commitment in our collaboration on this to the success of the > > "consumer" projects. So if the Arrow or Parquet communities were > > contemplating a change that might impact Kudu, for example, it would > > be in our best interest to be careful and communicate proactively. > > > > This all makes sense. From an Arrow and Parquet perspective, we do not > > add very much testing burden because our continuous integration suites > > do not take long to run. > > > > 3) Pre-commit/test mechanics > > > > One thing that would help would be community-maintained > > Dockerfiles/Docker images (or equivalent) to assist with validation > > and testing for developers. > > > > I am happy to comply with a pre-commit testing protocol that works for > > the Kudu and Impala teams. > > > > 4) Integration mechanics for breaking changes > > > >> One option is that each "user" of the libraries manually "rolls" to new > versions when they feel like it, but there's still now a case where a > common change "pushes work onto" the consumers to update call sites, etc. > > > > Breaking API changes will create extra work, because any automated > > testing that we create will not be able to validate the patch to the > > common library. Perhaps we can configure a manual way (in Jenkins, > > say) to test two patches together. > > > > In the event that a community member has a patch containing an API > > break that impacts a project that they are not a contributor for, > > there should be some expectation to either work with the affected > > project on a coordinated patch or obtain their +1 to merge the patch > > even though it will may require a follow up patch if the roll-forward > > in the consumer project exposes bugs in the common library. There may > > be situations like: > > > > * Kudu changes API in $COMMON that impacts Arrow > > * Arrow says +1, we will roll forward $COMMON later > > * Patch merged > > * Arrow rolls forward, discovers bug caused by patch in $COMMON > > * Arrow proposes patch to $COMMON > > * ... > > > > This is the worst case scenario, of course, but I actually think it is > > good because it would indicate that the unit testing in $COMMON needs > > to be improved. Unit testing in the common library, therefore, would > > take on more of a "defensive" quality than currently. > > > > In any case, I'm keen to move forward to coming up with a concrete > > plan if we can reach consensus on the particulars. > > > > Thanks > > Wes > > > > On Sun, Feb 26, 2017 at 10:18 PM, Leif Walsh <leif.wa...@gmail.com> > wrote: > >> I also support the idea of creating an "apache commons modern c++" style > >> library, maybe tailored toward the needs of columnar data processing > >> tools. I think APR is the wrong project but I think that *style* of > >> project is the right direction to aim. > >> > >> I agree this adds test and release process complexity across products > but I > >> think the benefits of a shared, well-tested library outweigh that, and > >> creating such test infrastructure will have long-term benefits as well. > >> > >> I'd be happy to lend a hand wherever it's needed. > >> > >> On Sun, Feb 26, 2017 at 4:03 PM Todd Lipcon <t...@cloudera.com> wrote: > >> > >>> Hey folks, > >>> > >>> As Henry mentioned, Impala is starting to share more code with Kudu > (most > >>> notably our RPC system, but that pulls in a fair bit of utility code as > >>> well), so we've been chatting periodically offline about the best way > to do > >>> this. Having more projects potentially interested in collaborating is > >>> definitely welcome, though I think does also increase the complexity of > >>> whatever solution we come up with. > >>> > >>> I think the potential benefits of collaboration are fairly > self-evident, so > >>> I'll focus on my concerns here, which somewhat echo Henry's. > >>> > >>> 1) Open source release model > >>> > >>> The ASF is very much against having projects which do not do releases. > So, > >>> if we were to create some new ASF project to hold this code, we'd be > >>> expected to do frequent releases thereof. Wes volunteered above to lead > >>> frequent releases, but we actually need at least 3 PMC members to vote > on > >>> each release, and given people can come and go, we'd probably need at > least > >>> 5-8 people who are actively committed to helping with the release > process > >>> of this "commons" project. > >>> > >>> Unlike our existing projects, which seem to release every 2-3 months, > if > >>> that, I think this one would have to release _much_ more frequently, > if we > >>> expect downstream projects to depend on released versions rather than > just > >>> pulling in some recent (or even trunk) git hash. Since the ASF > requires the > >>> normal voting period and process for every release, I don't think we > could > >>> do something like have "daily automatic releases", etc. > >>> > >>> We could probably campaign the ASF membership to treat this project > >>> differently, either as (a) a repository of code that never releases, in > >>> which case the "downstream" projects are responsible for vetting IP, > etc, > >>> as part of their own release processes, or (b) a project which does > >>> automatic releases voted upon by robots. I'm guessing that (a) is more > >>> palatable from an IP perspective, and also from the perspective of the > >>> downstream projects. > >>> > >>> > >>> 2) Governance/review model > >>> > >>> The more projects there are sharing this common code, the more > difficult it > >>> is to know whether a change would break something, or even whether a > change > >>> is considered desirable for all of the projects. I don't want to get > into > >>> some world where any change to a central library requires a multi-week > >>> proposal/design-doc/review across 3+ different groups of committers, > all of > >>> whom may have different near-term priorities. On the other hand, it > would > >>> be pretty frustrating if the week before we're trying to cut a Kudu > release > >>> branch, someone in another community decides to make a potentially > >>> destabilizing change to the RPC library. > >>> > >>> > >>> 3) Pre-commit/test mechanics > >>> > >>> Semi-related to the above: we currently feel pretty confident when we > make > >>> a change to a central library like kudu/util/thread.cc that nothing > broke > >>> because we run the full suite of Kudu tests. Of course the central > >>> libraries have some unit test coverage, but I wouldn't be confident > with > >>> any sort of model where shared code can change without verification by > a > >>> larger suite of tests. > >>> > >>> On the other hand, I also don't want to move to a model where any > change to > >>> shared code requires a 6+-hour precommit spanning several projects, > each of > >>> which may have its own set of potentially-flaky pre-commit tests, etc. > I > >>> can imagine that if an Arrow developer made some change to "thread.cc" > and > >>> saw that TabletServerStressTest failed their precommit, they'd have no > idea > >>> how to triage it, etc. That could be a strong disincentive to continued > >>> innovation in these areas of common code, which we'll need a good way > to > >>> avoid. > >>> > >>> I think some of the above could be ameliorated with really good > >>> infrastructure -- eg on a test failure, automatically re-run the failed > >>> test on both pre-patch and post-patch, do a t-test to check statistical > >>> significance in flakiness level, etc. But, that's a lot of > infrastructure > >>> that doesn't currently exist. > >>> > >>> > >>> 4) Integration mechanics for breaking changes > >>> > >>> Currently these common libraries are treated as components of > monolithic > >>> projects. That means it's no extra overhead for us to make some kind of > >>> change which breaks an API in src/kudu/util/ and at the same time > updates > >>> all call sites. The internal libraries have no semblance of API > >>> compatibility guarantees, etc, and adding one is not without cost. > >>> > >>> Before sharing code, we should figure out how exactly we'll manage the > >>> cases where we want to make some change in a common library that > breaks an > >>> API used by other projects, given there's no way to make an atomic > commit > >>> across many repositories. One option is that each "user" of the > libraries > >>> manually "rolls" to new versions when they feel like it, but there's > still > >>> now a case where a common change "pushes work onto" the consumers to > update > >>> call sites, etc. > >>> > >>> Admittedly, the number of breaking API changes in these common > libraries is > >>> relatively small, but would still be good to understand how we would > plan > >>> to manage them. > >>> > >>> -Todd > >>> > >>> On Sun, Feb 26, 2017 at 10:12 AM, Wes McKinney <wesmck...@gmail.com> > >>> wrote: > >>> > >>>> hi Henry, > >>>> > >>>> Thank you for these comments. > >>>> > >>>> I think having a kind of "Apache Commons for [Modern] C++" would be an > >>>> ideal (though perhaps initially more labor intensive) solution. > >>>> There's code in Arrow that I would move into this project if it > >>>> existed. I am happy to help make this happen if there is interest from > >>>> the Kudu and Impala communities. I am not sure logistically what would > >>>> be the most expedient way to establish the project, whether as an ASF > >>>> Incubator project or possibly as a new TLP that could be created by > >>>> spinning IP out of Apache Kudu. > >>>> > >>>> I'm interested to hear the opinions of others, and possible next > steps. > >>>> > >>>> Thanks > >>>> Wes > >>>> > >>>> On Sun, Feb 26, 2017 at 2:12 AM, Henry Robinson <he...@apache.org> > >>> wrote: > >>>>> Thanks for bringing this up, Wes. > >>>>> > >>>>> On 25 February 2017 at 14:18, Wes McKinney <wesmck...@gmail.com> > >>> wrote: > >>>>> > >>>>>> Dear Apache Kudu and Apache Impala (incubating) communities, > >>>>>> > >>>>>> (I'm not sure the best way to have a cross-list discussion, so I > >>>>>> apologize if this does not work well) > >>>>>> > >>>>>> On the recent Apache Parquet sync call, we discussed C++ code > sharing > >>>>>> between the codebases in Apache Arrow and Apache Parquet, and > >>>>>> opportunities for more code sharing with Kudu and Impala as well. > >>>>>> > >>>>>> As context > >>>>>> > >>>>>> * We have an RC out for the 1.0.0 release of apache-parquet-cpp, the > >>>>>> first C++ release within Apache Parquet. I got involved with this > >>>>>> project a little over a year ago and was faced with the unpleasant > >>>>>> decision to copy and paste a significant amount of code out of > >>>>>> Impala's codebase to bootstrap the project. > >>>>>> > >>>>>> * In parallel, we begin the Apache Arrow project, which is designed > to > >>>>>> be a complementary library for file formats (like Parquet), storage > >>>>>> engines (like Kudu), and compute engines (like Impala and pandas). > >>>>>> > >>>>>> * As Arrow and parquet-cpp matured, an increasing amount of code > >>>>>> overlap crept up surrounding buffer memory management and IO > >>>>>> interface. We recently decided in PARQUET-818 > >>>>>> (https://github.com/apache/parquet-cpp/commit/ > >>>>>> 2154e873d5aa7280314189a2683fb1e12a590c02) > >>>>>> to remove some of the obvious code overlap in Parquet and make > >>>>>> libarrow.a/so a hard compile and link-time dependency for > >>>>>> libparquet.a/so. > >>>>>> > >>>>>> * There is still quite a bit of code in parquet-cpp that would > better > >>>>>> fit in Arrow: SIMD hash utilities, RLE encoding, dictionary > encoding, > >>>>>> compression, bit utilities, and so forth. Much of this code > originated > >>>>>> from Impala > >>>>>> > >>>>>> This brings me to a next set of points: > >>>>>> > >>>>>> * parquet-cpp contains quite a bit of code that was extracted from > >>>>>> Impala. This is mostly self-contained in > >>>>>> https://github.com/apache/parquet-cpp/tree/master/src/parquet/util > >>>>>> > >>>>>> * My understanding is that Kudu extracted certain computational > >>>>>> utilities from Impala in its early days, but these tools have likely > >>>>>> diverged as the needs of the projects have evolved. > >>>>>> > >>>>>> Since all of these projects are quite different in their end goals > >>>>>> (runtime systems vs. libraries), touching code that is tightly > coupled > >>>>>> to either Kudu or Impala's runtimes is probably not worth > discussing. > >>>>>> However, I think there is a strong basis for collaboration on > >>>>>> computational utilities and vectorized array processing. Some > obvious > >>>>>> areas that come to mind: > >>>>>> > >>>>>> * SIMD utilities (for hashing or processing of preallocated > contiguous > >>>>>> memory) > >>>>>> * Array encoding utilities: RLE / Dictionary, etc. > >>>>>> * Bit manipulation (packing and unpacking, e.g. Daniel Lemire > >>>>>> contributed a patch to parquet-cpp around this) > >>>>>> * Date and time utilities > >>>>>> * Compression utilities > >>>>>> > >>>>> > >>>>> Between Kudu and Impala (at least) there are many more opportunities > >>> for > >>>>> sharing. Threads, logging, metrics, concurrent primitives - the list > is > >>>>> quite long. > >>>>> > >>>>> > >>>>>> > >>>>>> I hope the benefits are obvious: consolidating efforts on unit > >>>>>> testing, benchmarking, performance optimizations, continuous > >>>>>> integration, and platform compatibility. > >>>>>> > >>>>>> Logistically speaking, one possible avenue might be to use Apache > >>>>>> Arrow as the place to assemble this code. Its thirdparty toolchain > is > >>>>>> small, and it builds and installs fast. It is intended as a library > to > >>>>>> have its headers used and linked against other applications. (As an > >>>>>> aside, I'm very interested in building optional support for Arrow > >>>>>> columnar messages into the kudu client). > >>>>>> > >>>>> > >>>>> In principle I'm in favour of code sharing, and it seems very much in > >>>>> keeping with the Apache way. However, practically speaking I'm of the > >>>>> opinion that it only makes sense to house shared support code in a > >>>>> separate, dedicated project. > >>>>> > >>>>> Embedding the shared libraries in, e.g., Arrow naturally limits the > >>> scope > >>>>> of sharing to utilities that Arrow is interested in. It would make no > >>>> sense > >>>>> to add a threading library to Arrow if it was never used natively. > >>>> Muddying > >>>>> the waters of the project's charter seems likely to lead to user, and > >>>>> developer, confusion. Similarly, we should not necessarily couple > >>> Arrow's > >>>>> design goals to those it inherits from Kudu and Impala's source code. > >>>>> > >>>>> I think I'd rather see a new Apache project than re-use a current one > >>> for > >>>>> two independent purposes. > >>>>> > >>>>> > >>>>>> > >>>>>> The downside of code sharing, which may have prevented it so far, > are > >>>>>> the logistics of coordinating ASF release cycles and keeping build > >>>>>> toolchains in sync. It's taken us the past year to stabilize the > >>>>>> design of Arrow for its intended use cases, so at this point if we > >>>>>> went down this road I would be OK with helping the community commit > to > >>>>>> a regular release cadence that would be faster than Impala, Kudu, > and > >>>>>> Parquet's respective release cadences. Since members of the Kudu and > >>>>>> Impala PMC are also on the Arrow PMC, I trust we would be able to > >>>>>> collaborate to each other's mutual benefit and success. > >>>>>> > >>>>>> Note that Arrow does not throw C++ exceptions and similarly follows > >>>>>> Google C++ style guide to the same extent at Kudu and Impala. > >>>>>> > >>>>>> If this is something that either the Kudu or Impala communities > would > >>>>>> like to pursue in earnest, I would be happy to work with you on next > >>>>>> steps. I would suggest that we start with something small so that we > >>>>>> could address the necessary build toolchain changes, and develop a > >>>>>> workflow for moving around code and tests, a protocol for code > reviews > >>>>>> (e.g. Gerrit), and coordinating ASF releases. > >>>>>> > >>>>> > >>>>> I think, if I'm reading this correctly, that you're assuming > >>> integration > >>>>> with the 'downstream' projects (e.g. Impala and Kudu) would be done > via > >>>>> their toolchains. For something as fast moving as utility code - and > >>>>> critical, where you want the latency between adding a fix and > including > >>>> it > >>>>> in your build to be ~0 - that's a non-starter to me, at least with > how > >>>> the > >>>>> toolchains are currently realised. > >>>>> > >>>>> I'd rather have the source code directly imported into Impala's tree > - > >>>>> whether by git submodule or other mechanism. That way the coupling is > >>>>> looser, and we can move more quickly. I think that's important to > other > >>>>> projects as well. > >>>>> > >>>>> Henry > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> Let me know what you think. > >>>>>> > >>>>>> best > >>>>>> Wes > >>>>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> Todd Lipcon > >>> Software Engineer, Cloudera > >>> > >> -- > >> -- > >> Cheers, > >> Leif > > -- -- Cheers, Leif
