And of course I meant partitions of a dataset in the same machine... Btw it is mentioned in our storage paper. "Currently in AsterixDB, all partitions of any one dataset and its indexes in a machine share the same memory budget for in-memory components (divided equally across the indexes). This implies that when the memory component of index in a partition is declared to be full, all memory components of the indexes for the same dataset in that machine are also declared to be full. Therefore, multiple consecutive flush requests are sent to the I/O scheduler for flushing the memory components of that dataset and its indexes for all the partitions located on the machine.“
Sattam On Oct 24, 2017 6:12 AM, "Sattam Alsubaiee" <[email protected]> wrote: > There is a fundamental reason behind that (unless that has changed). All > partitions of a dataset (primary and secondary indexes) share the memory > budget for their in-memory components. So they use the same optracker to > synchronize their operations. > > > Sattam > > On Oct 24, 2017 5:59 AM, "Wail Alkowaileet" <[email protected]> wrote: > >> Thanks Abdullah! >> >> On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi <[email protected]> >> wrote: >> >> > Hi Wail, >> > There is no fundamental reason why it is one. In fact, it has been on >> our >> > todo for a long time to make it one per partition. >> > >> > Cheers, >> > Abdullah. >> > >> > > On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <[email protected]> >> > wrote: >> > > >> > > Dear devs, >> > > >> > > I have a question regarding the opTracker. Currently, we initialize >> one >> > > opTracker per dataset in every NC. >> > > >> > > My question is why it's per dataset not per partition ? Is there a >> > > transactional constraints for that ? >> > > >> > > From what I can see that the opTracker can create a lot of contention >> in >> > > case there're many IO devices. For instance, each insert will call >> > > *LSMHarness.getAndEnterComponents()* [1], which >> > *synchronize(opTracker). *That >> > > means (correct me if I'm wrong), insert is going to serialize the >> > > *enterComponent()* part among partitions. >> > > >> > > [1] >> > > https://github.com/apache/asterixdb/blob/master/hyracks- >> > fullstack/hyracks/hyracks-storage-am-lsm-common/src/ >> > main/java/org/apache/hyracks/storage/am/lsm/common/impls/ >> > LSMHarness.java#L86 >> > > >> > > -- >> > > >> > > *Regards,* >> > > Wail Alkowaileet >> > >> > >> >> >> -- >> >> *Regards,* >> Wail Alkowaileet >> >
