[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156630#comment-16156630 ] stack commented on HBASE-18446: --- I wrote the pheonix dev list on CPs in general but asking for input on this issue in particular. +1 on proceeding. Thanks [~Apache9]. Above sounds good. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0-alpha-3 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156292#comment-16156292 ] Duo Zhang commented on HBASE-18446: --- If no objections I will provide a patch soon. We need to decide the final coprocessor API before beta release I think. Thanks. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0-alpha-3 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152440#comment-16152440 ] Duo Zhang commented on HBASE-18446: --- Let's revive this. So I think we all agree that user should not have the ability to create a StoreFile instance, but for now Phoenix still need to extend StoreFileScanner to support secondary index. So let's mark StoreFileScanner change the comments of preBulkLoadHFile? And also, mark StoreFileScanner as IA.LimitedPrivate(Phoenix)? Thanks. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0-alpha-3 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[
https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116177#comment-16116177
]
Anoop Sam John commented on HBASE-18446:
Regarding HBASE-6224 and load files in CP itself.
I don't think any one would be doing that. At that time we had a design of all
pre hooks can do a bypass on the Context so the actual op wont be done. Similar
way this was also done I guess. Because u can see in RSRpcServices
{code}
if (region.getCoprocessorHost() != null) {
bypass = region.getCoprocessorHost().preBulkLoadHFile(familyPaths);
}
try {
if (!bypass) {
map = region.bulkLoadHFiles(familyPaths, request.getAssignSeqNum(),
null,
request.getCopyFile());
if (map != null) {
loaded = true;
}
}
...
builder.setLoaded(loaded);
return builder.build();
{code}
When bypass is done, we set loaded as false only. The return to the called code
will indicate the bulk load being NOT done. The intent of that old jira was to
add the pre hook so that AC can check and if failed it would throw Exception
only. May be we can consider this op as some thing on which bypass is not
possible! (Like preClose)
> Mark StoreFileScanner as IA.Private
> ---
>
> Key: HBASE-18446
> URL: https://issues.apache.org/jira/browse/HBASE-18446
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Duo Zhang
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
>
> Do not see any reason why it is marked as IA.LimitedPrivate. It is not
> referenced in any CPs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104881#comment-16104881 ] Duo Zhang commented on HBASE-18446: --- Oh, seems no problem. The compaction will also use the replaced StoreFileReader so it can read the index and write them to the new StoreFile. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104876#comment-16104876 ] Anoop Sam John commented on HBASE-18446: For the compaction purpose also, each of the region's compaction would have selected the corresponding Half file and its reader for doing the compaction work (Scan for compaction). As of 2.0 design, we wont archive the compacted away files immediately after the compaction. The old scans will continue using them. See CompactedHFilesDischarger. The new compacted files will have proper data as per the split daughter regions > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[
https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104871#comment-16104871
]
Duo Zhang commented on HBASE-18446:
---
{quote}
but with local index we need to do full HFile scan and should be able to find
the whether Cell belongs the child region based on actual data row key
{quote}
What happens if the file is compacted?
> Mark StoreFileScanner as IA.Private
> ---
>
> Key: HBASE-18446
> URL: https://issues.apache.org/jira/browse/HBASE-18446
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Duo Zhang
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
>
> Do not see any reason why it is marked as IA.LimitedPrivate. It is not
> referenced in any CPs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104860#comment-16104860 ] Anoop Sam John commented on HBASE-18446: Ya then StoreFileReader is what should be exposed. As Duo said not the impl class but an interface. Then why expose StoreFileScanner and StoreFile? > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[
https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104677#comment-16104677
]
Duo Zhang commented on HBASE-18446:
---
OK, got it. Seems we need to introduce an interface for {{StoreFileReader}} if
we want to hide the implementation details to CP users. A big refactoring...
> Mark StoreFileScanner as IA.Private
> ---
>
> Key: HBASE-18446
> URL: https://issues.apache.org/jira/browse/HBASE-18446
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Duo Zhang
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
>
> Do not see any reason why it is marked as IA.LimitedPrivate. It is not
> referenced in any CPs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104479#comment-16104479 ] Rajeshbabu Chintaguntla commented on HBASE-18446: - [~Apache9] [~anoop.hbase] We use LocalIndexStoreFileScanner, IndexHalfStoreFileReader to implement HalfStoreFileReader functionality supported after region split/merges. With HalfStoreFileReader we limit the HFile scan to the child region boundary but with local index we need to do full HFile scan and should be able to find the whether Cell belongs the child region based on actual data row key. We use preStoreFileReaderOpen cp hook to create the custom reader one time. Since the hook returns the reader we need to just extend it instead of wrapping into some other class. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104416#comment-16104416 ] Anoop Sam John commented on HBASE-18446: [~chrajeshbab...@gmail.com] Can you tell us the exact needs and usage of StoreFileReader in local index? Sorry I forgot those details totally. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[
https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104314#comment-16104314
]
Duo Zhang commented on HBASE-18446:
---
OK, seems there are lots of conflicts...{{StoreFileReader}} is marked as
IA.Private when it was extracted out from StoreFile...
And it is really a pain that coprocessor users need to extend these
implementation classes. [~chrajeshbab...@gmail.com] and [~samarthjain], is it
possible to implement the index based StoreFileReader by wrapping the original
StoreFileReader other than extending it?
Thanks.
> Mark StoreFileScanner as IA.Private
> ---
>
> Key: HBASE-18446
> URL: https://issues.apache.org/jira/browse/HBASE-18446
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Duo Zhang
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
>
> Do not see any reason why it is marked as IA.LimitedPrivate. It is not
> referenced in any CPs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103482#comment-16103482 ] Andrew Purtell commented on HBASE-18446: I don't know of any use cases for the preBulkLoadHFile hook itself. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103480#comment-16103480 ] Andrew Purtell commented on HBASE-18446: [~Apache9] Phoenix uses StoreFile and StoreFileReader in the local index implementation. phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReaderGenerator.java phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReader.java phoenix-core/src/main/java/org/apache/phoenix/util/RepairUtil.java So I think we need to expose these as LimitedPrivate. We could do more and make these into reduced and supportable interfaces like we did when introducing the Region interface. There are other references, but they are just treating the types as opaque, passing them from a CP hook to a delegate method etc. [~rajeshbabu] [~samarthjain] > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[
https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103197#comment-16103197
]
Duo Zhang commented on HBASE-18446:
---
OK, we have this
{code:title=RegionObserver.java}
/**
* Called before bulkLoadHFile. Users can create a StoreFile instance to
* access the contents of a HFile.
*
* @param ctx the environment provided by the region server
* @param familyPaths pairs of { CF, HFile path } submitted for bulk load.
Adding
* or removing from this list will add or remove HFiles to be bulk loaded.
*/
default void preBulkLoadHFile(final
ObserverContext ctx,
List> familyPaths) throws IOException {}
{code}
[~apurtell] Sir do you have any use cases of this method? I think we can
provide a {{createStoreFile}} method in the {{Region}} interface to avoid users
use the implementation class directly. But I do not know how do users use the
{{StoreFile}} instance. And on 'access the contents of a HFile', this means we
need to also expose {{StoreFileReader}} to users? Oh no, it is also an
implementation class...
We need to hurry up...
Thanks.
> Mark StoreFileScanner as IA.Private
> ---
>
> Key: HBASE-18446
> URL: https://issues.apache.org/jira/browse/HBASE-18446
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Duo Zhang
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
>
> Do not see any reason why it is marked as IA.LimitedPrivate. It is not
> referenced in any CPs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18446) Mark StoreFileScanner as IA.Private
[ https://issues.apache.org/jira/browse/HBASE-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101225#comment-16101225 ] Duo Zhang commented on HBASE-18446: --- OK, I think it is bit complicated here. In HBASE-6224, seems the intention is that in preBulkLoadHFile we can bypass the default load operation and create the StoreFiles and then load it by ourselves. This is impossible after we change StoreFile to an interface. We need to find another way to do the same thing. > Mark StoreFileScanner as IA.Private > --- > > Key: HBASE-18446 > URL: https://issues.apache.org/jira/browse/HBASE-18446 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > > Do not see any reason why it is marked as IA.LimitedPrivate. It is not > referenced in any CPs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
