Do you mean the modifications to the metamx/druid-spark-batch project or the new ShardSpec you're working on?
On Thu, Jan 10, 2019 at 3:09 PM Julian Jaffe <jja...@pinterest.com.invalid> wrote: > Thanks for the detailed pointers, Gian! In light of the ongoing discussion > around on-list development, does this seem like something that's worthwhile > to anyone else in the community? > > On Tue, Jan 8, 2019 at 10:32 AM Gian Merlino <g...@apache.org> wrote: > > > Hey Julian, > > > > There aren't any gotchas that I can think of other than the fact that > they > > are not super well documented, and you might miss some features if you're > > just skimming the code. A couple points that might matter, > > > > 1) PartitionChunk is what allows a shard spec to contribute to the > > definition of whether a set of segments for a time chunk is "complete". > > It's an important concept since the broker will not query segment sets > > unless the chunk is complete. The way the completeness check works is > > basically that the broker will get all the ShardSpecs for all the > segments > > in a time chunk, order them by partitionNum, generate the partition > chunks, > > and check if (a) the first one is a starter based on "isStart", (b) any > > subsequent ones until the end [based on "isEnd"] abut the previous one > > [based on "abuts"]. Some ShardSpecs use nonsensical-at-first-glance logic > > from these methods to short circuit the completeness checks: time chunks > > with LinearShardSpecs are _always_ considered complete. Time chunks with > > NumberedShardSpecs can have "partitionNum" go beyond "partitions", and > are > > considered complete if the first "partitions" number of segments are > > present. > > > > 2) "getDomainDimensions" and "possibleInDomain" are optional, but useful > > for powering broker-side segment pruning. > > > > 3) All segments in the same time chunk must have the same kind of > > ShardSpec. However, it can vary from time chunk to time chunk within a > > datasource. > > > > On Mon, Jan 7, 2019 at 2:56 PM Julian Jaffe <jja...@pinterest.com.invalid > > > > wrote: > > > > > Hey all, > > > > > > Are there any major caveats or gotchas I should be aware of when > > > implementing a new ShardSpec? The context here is that we have a > > datasource > > > that is the combined result of multiple input jobs. We're trying to do > > > write-side joining by having all of the jobs write segments for the > same > > > intervals (e.g. partitioning on both partition number and source > > pipeline). > > > For now, I've modified the Spark-Druid batch ingestor ( > > > https://github.com/metamx/druid-spark-batch) to run in our various > > > pipelines and to write out segments with identifier form > > > `dataSource_startInterval_endInterval_version_sourceName_partitionNum. > > This > > > is working without issue for loading, querying, and deleting data, but > > the > > > metadata API reports the incorrect segment identifier, since it > > > reconstructs the identifier instead of reading from metadata (e.g. it > > > reports segment identifiers of the form > > > `dataSource_startInterval_endInterval_version_partitionNum`). Both > > because > > > we'd like this to be fully supported, and because we imagine that this > > > feature may be useful to others, I'd like to implement this via a > > > ShardSpec. > > > > > > Julian > > > > > >
