Actually, it's worse than that... We have more than one key -> partition mapping for the same key (part of a CQRS pattern we use).
Aren't key affinity functions essentially an API in any event? Cheers, Raymond. On Wed, Nov 4, 2020 at 9:54 PM Valentin Kulichenko < [email protected]> wrote: > Thanks, Raymond. So the reason why you couldn't use the @AffinityKeyMapped > annotation or the CacheKeyConfiguration is that collocation is based on > *two fields*, not just one field. Is my understanding correct? > > If that's the case, I believe it can be easily improved by providing ways > to specify multiple fields for collocation. The fact that this scenario > requires a custom affinity mapper is a usability issue in my view. > > -Val > > On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <[email protected] > > > wrote: > > > If I have a primary key vector like this <x><y><t>, where <x> & <y> can > > have varying values for <t>, and I want all keys with the same <x> and > <y> > > to reside in the same partition for processing colocation requirements > then > > I can't use the standard Ignite mapping to do this, I need to use a > custom > > mapper than uses just the <x> & <y> components. > > > > > > On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko < > > [email protected]> wrote: > > > > > Raymond, > > > > > > Thanks for the details. So it sounds like you have a custom affinity > > > mapper, not affinity function. This makes things simpler, but I'm still > > > failing to understand why standard mechanisms for collocation [1] > didn't > > > work for you. Could you please clarify? > > > > > > [1] > > > > https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation > > > > > > -Val > > > > > > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson < > > [email protected] > > > > > > > wrote: > > > > > > > In terms of Key -> Partition -> Node mapping, we provide customer > > > affinity > > > > mappers for the Key -> Partition stage an allow Ignite to map > > partitions > > > to > > > > nodes. > > > > > > > > Our keys are structs with multiple fields forming a composite primary > > > key, > > > > parts of which are spatially identifying and parts contain other > > > > characteristics. We want to ensure all related spatial elements > resolve > > > to > > > > the same partition and so will be placed on the same node for query > > > > colocation purposes. > > > > > > > > > > > > > > > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko < > > > > [email protected]> wrote: > > > > > > > > > Moti, Raymond, > > > > > > > > > > Could you please describe your use cases in more detail? What are > the > > > > types > > > > > used as cache keys? What is the custom logic that you use for > > affinity > > > > > mapping? What was the exact reason to customize versus using > built-in > > > > > collocation mechanisms? > > > > > > > > > > Ultimately, I'm sure that custom affinity functions will go away > > cause > > > > they > > > > > introduce multiple potential issues. However, I would definitely > like > > > to > > > > > make sure that your use cases are still supported in Ignite 3.0 via > > > some > > > > > other means. > > > > > > > > > > -Val > > > > > > > > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov < > > > > > [email protected]> wrote: > > > > > > > > > > > Thanks for the clarification. > > > > > > > > > > > > There was no intention to remove the customizable key to > partition > > > > > mapping. > > > > > > > > > > > > Difficulties arise when mapping partitions to nodes, so it's > > > desirable > > > > to > > > > > > have internally tested implementation with a way to customize > it's > > > > > behavior > > > > > > without additional coding on the user side. > > > > > > > > > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson < > > > > [email protected] > > > > > >: > > > > > > > > > > > > > Just to be clear, the affinity functions we are using convert > > keys > > > to > > > > > > > partitions, we do not map partitions to nodes and leave that to > > > > Ignite. > > > > > > > > > > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hello. > > > > > > > > > > > > > > > > Custom affinity functions can cause weird bugs and data loss > if > > > > > > > implemented > > > > > > > > wrongly. > > > > > > > > There is an intention to keep a backup filter based on user > > > > > attributes > > > > > > > > (with additional validation logic to ensure correctness) for > > > > > > controllable > > > > > > > > data placement. > > > > > > > > > > > > > > > > Can you describe more precisely why you had to implement > custom > > > > > > affinity > > > > > > > > functions and not resort to default rendezvous affinity + > > backup > > > > > > filter ? > > > > > > > > > > > > > > > > > > > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson < > > > > > > [email protected] > > > > > > > >: > > > > > > > > > > > > > > > > > We also use custom affinity functions (vis the C# client). > > > > > > > > > > > > > > > > > > The wish list mentions use of a particular annotation > > > > > > > > > (@CentralizedAffinityFunction): > > > > > > > > > Is the wish to remove just this annotation, or the ability > to > > > > > define > > > > > > > > custom > > > > > > > > > affinity functions at all? > > > > > > > > > > > > > > > > > > In our case we use affinity functions to ensure particular > > > > > > distribution > > > > > > > > of > > > > > > > > > spatial data across a processing cluster to ensure even > load > > > > > > > management. > > > > > > > > > > > > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I saw at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist > > > > > > > > > > that custom affinity functions are on the potential > > wishlist > > > > for > > > > > > > > removal. > > > > > > > > > > The way we're using it's very critical that we be able to > > > > control > > > > > > the > > > > > > > > > > placement of data quite precisely - as part of that we > > > specify > > > > > > > > explicitly > > > > > > > > > > the partition we want in the key, and then our affinity > > > > function > > > > > > uses > > > > > > > > > that > > > > > > > > > > (else delegating to default rendezvous). We don't need > all > > > the > > > > > > > > > > abilities there, although I think that often others do. > > > > > > > > > > > > > > > > > > > > This seems to me to be a case that the benefit of > removing > > > this > > > > > is > > > > > > > > > minimal > > > > > > > > > > and could cause quite a lot of disruption to users. > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > <http://www.trimble.com/> > > > > > > > > > Raymond Wilson > > > > > > > > > Solution Architect, Civil Construction Software Systems > > (CCSS) > > > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand > > > > > > > > > +64-21-2013317 Mobile > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Alexei Scherbakov > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > <http://www.trimble.com/> > > > > > > > Raymond Wilson > > > > > > > Solution Architect, Civil Construction Software Systems (CCSS) > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand > > > > > > > +64-21-2013317 Mobile > > > > > > > [email protected] > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Best regards, > > > > > > Alexei Scherbakov > > > > > > > > > > > > > > > > > > > > > > > -- > > > > <http://www.trimble.com/> > > > > Raymond Wilson > > > > Solution Architect, Civil Construction Software Systems (CCSS) > > > > 11 Birmingham Drive | Christchurch, New Zealand > > > > +64-21-2013317 Mobile > > > > [email protected] > > > > > > > > < > > > > > > > > > > https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch > > > > > > > > > > > > > > > > > > -- > > <http://www.trimble.com/> > > Raymond Wilson > > Solution Architect, Civil Construction Software Systems (CCSS) > > 11 Birmingham Drive | Christchurch, New Zealand > > +64-21-2013317 Mobile > > [email protected] > > > > < > > > https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch > > > > > > -- <http://www.trimble.com/> Raymond Wilson Solution Architect, Civil Construction Software Systems (CCSS) 11 Birmingham Drive | Christchurch, New Zealand +64-21-2013317 Mobile [email protected] <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>
