[
https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507594#comment-13507594
]
Sergey Shelukhin commented on HBASE-7236:
-----------------------------------------
[~stack]
bq. IIUC, you are expanding CompoundConfiguration to do table and 'overrides'?
Currently it already has metadata, I am adding overrides. The method name is
special only because Java won't allow two method differing only by typename
part of the parameter. It's no different than other "add"-s :)
bq. How would this help us get to changing configuration on the fly? Looks like
it doesn't change our current story. CompoundConfiguration is setup in HRegion
or ColumnFamily setup still..
It doesn't (well, other than setting overrides after disabling the table, which
is more dynamic than XML config replacement).
For XML config, there's HBASE-3909 to solve that problem; for column updates,
I'd prefer to make online-alter "bulletproof" - that way we'd get this config
to be fully dynamic as a side benefit.
Two ideas we had was tracking schema version on master and server, and
validating on master what server has, but that's inconvenient right now given
how it's stored, and may lead to other race conditions; or using new 2pc-ish
barrier mechanism that is being introduced for snapshots, somehow.
Regardless, in this JIRA I do not aim to solve this problem...
bq. If we start to record metadata on a table, say column types, would we use
this mechanism?
You mean CompoundConfiguration mechanism, or overrides mechanism?
Not overrides (they are intended for config), but as CompoundConfiguration is
already used for cf metadata, I don't see why not use it also for table
metadata if appropriate.
bq. How would 'overrides' be specified in the shell say? (Patch doesn't seem to
say) We have means of altering table and column descriptors. Where would
'overrides' go?
Similarly to metadata. I am going to rename "CONFIG" to something indicating
it's actually user metadata, and use name like CONFIGURATION_OVERRIDES - a
separate Ruby hash with values.
Didn't want to go thru it before confirmation on the viability of this approach
:)
bq. bq. however, making it explicit and separate from miscellaneous metadata
would be cleaner imho.
bq. Can you say more on above?
See below.
bq. HTableDescriptor and HColumnDescriptor dictionaries are key/value maps that
get persisted into the filesystem when changed. We read them them throughout
the codebase and we list them in master UIs, etc. Will they blow up under this
new use case? HTD and HCD are mostly schema with a little config. This
direction would seem to be using these descriptors to add table or column
scoped configs. Should we be working to undo schema and config conflation
rather than compound it?
They may blow up (depending on how much config one wants to override), this is
one of the reasons I want to keep it separate, to have finer grained control
for things like UI or shell.
Do you suggest we keep them even more separate, e.g. separate
class/serialization mechanism? It seems it would add complexity, esp. given
that some of the existing METADATA is config additions/overrides with custom
names, essentially.
Can you elaborate on example mechanism (e.g. meta-like table for tables/cfs,
different file from tableinfo, ...)?
The other thing about the current map is that it's already multi-purpose (see
HBASE-7237; in the discussion in the shell issue when I wanted to keep CONFIG
separate from system stuff I didn't realize they were still stored in the same
map in HTD).
Dumping config overrides there in addition to system metadata and user metadata
is inelegant imho.
Finally, the third thing is that if it's separate we can have stricter
validation - e.g. whitelist what can and cannot be overridden, or even validate
values.
> add per-table/per-cf compaction configuration via metadata
> ----------------------------------------------------------
>
> Key: HBASE-7236
> URL: https://issues.apache.org/jira/browse/HBASE-7236
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
> Affects Versions: 0.96.0
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Attachments: HBASE-7236-PROTOTYPE.patch, HBASE-7236-PROTOTYPE.patch
>
>
> Regardless of the compaction policy, it makes sense to have separate
> configuration for compactions for different tables and column families, as
> their access patterns and workloads can be different. In particular, for
> tiered compactions that are being ported from 0.89-fb branch it is necessary
> to have, to use it properly.
> We might want to add support for compaction configuration via metadata on
> table/cf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
