Maybe the master is only tracking sizes in mb because that’s what is sent from the region servers via heartbeat protocol? Reducing the load over the wire makes sense as a scalability concern.
On Wed, Oct 13, 2021 at 03:45 Norbert Kalmar <[email protected]> wrote: > Thank you for the inputs. > Yes, we do return in bytes, but it is multiplied from MB so we get a false > size of 0 below 1MB. > > I checked HBASE-16169, even before that patch we used MB in > REgionSizeCalculator, the patch just kept the idea of using MB as a unit. > > I will map every usage related to region size and try to figure out the > reason why MB is the standardize unit. And after all, we can always convert > to MB - as we do convert to Bytes right now. But this way we wouldn't lose > the precise size due to conversation if we need the size in Bytes. > > I'll update my PR once I think I figured out the solution. > > - Norbert > > > > On Wed, Oct 13, 2021 at 4:59 AM 张铎(Duo Zhang) <[email protected]> > wrote: > > > The return value is in bytes, the problem is that we normalize the size > in > > MB and then multiply MB to get the size in bytes, so if a file is less > than > > 1MB, the returned value will be zero. > > > > Need to investigate more here. > > > > Reading the issue, the scalable problem they wanted to solve is that we > > will go to master to get the region size, not about whether the unit is > in > > MB or not. > > > > Thanks. > > > > Nick Dimiduk <[email protected]> 于2021年10月13日周三 上午7:47写道： > > > > > Hi Norbert, > > > > > > To answer your question directly: the RegionSizeCalculator class is > > > annotated with @InterfaceAudience.Private, which means there's a good > > > chance that it's implementation can be changed without need for a > > > deprecation cycle and user participation. > > > > > > Curiously, I noticed that this `sizeMap` is accessed down in the method > > > `long getRegionSize(byte[])`, and its javadoc mentions the returned > unit > > > explicitly as bytes. > > > > > > So with a little investigation using git blame, I see that the switch > > from > > > returning values in bytes to values in megabytes came in through > > > HBASE-16169 -- your proposed change was the old implementation. For > > > whatever reasons, it was determined to not be scalable. So, we could > > revert > > > back, but we'd need some new solution to what HBASE-16169 aimed to > solve. > > > > > > I hope this helps. > > > > > > Thanks, > > > Nick > > > > > > On Tue, Oct 12, 2021 at 10:54 AM Norbert Kalmar <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > There is a new optimization in spark (SPARK-34809) where > > > ignoreEmptySplits > > > > filters out all regions that's size is 0. They use a hadoop library > > > > getSize() in TableInputFormat. > > > > > > > > Drilling down, this will return Bytes, but it converts it from > > MegaBytes > > > - > > > > meaning anything under 1 MB will come down as 0 Bytes, meaning empty. > > > > I did a quick PR I thought would help: > > > > https://github.com/apache/hbase/pull/3737 > > > > But it turns out it's not as easy as requesting the size in Bytes > > instead > > > > of MB from Size class, as we set it in MB te begin with in > > > > RegionMetricsBuilder > > > > -> setStoreFileSize(new Size(regionLoadPB.getStorefileSizeMB(), > > > > Size.Unit.MEGABYTE)) > > > > > > > > I did some testing, and inserting a few kilobytes of data, then > > > > calling list_regions > > > > will in fact give back size 0. > > > > > > > > My question is, is it okay to store the region size in Bytes instead? > > > > Mainly asking because of backward compatibility reasons. > > > > > > > > Regards, > > > > Norbert > > > > > > > > > >
