[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991977#comment-14991977 ] Apekshit Sharma commented on HBASE-14769: - HBASE-9182 added getTableNames(). But the reasons I think it's okay to remove all these functions are, these are all unused String/byte[] counterparts of TableName and do not need to be marked deprecated because the class was marked Audience private in 1.0.0. I am not removing any functions which are still required to support Admin class' deprecated functions. Those can be removed in 3.0.0. Thanks [~stack] for reviewing. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991986#comment-14991986 ] Apekshit Sharma commented on HBASE-14769: - Will look into ruby code and fix hudson errors. Thanks [~sduskis]. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12072) Standardize retry handling for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992342#comment-14992342
]
Apekshit Sharma commented on HBASE-12072:
-
You're right, i should have seen the pattern and corrected myself on seeing the
same thing in 3 jiras.
To confirm, we add unreleased version corresponding to every brach the change
was pushed, right?
If yes, seems like 0.99.2 was made from 1.0 branch.
Also let me know if I should revert the changes i made.
> Standardize retry handling for master operations
>
>
> Key: HBASE-12072
> URL: https://issues.apache.org/jira/browse/HBASE-12072
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 1.0.0, 2.0.0, 0.99.2
>
> Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
> hbase-12072_v2.patch, hbase-12072_v2.patch, hbase-12072_v3.patch
>
>
> For master requests, there are two retry mechanisms in effect. The first one
> is from HBaseAdmin.executeCallable()
> {code}
> private V executeCallable(MasterCallable callable) throws
> IOException {
> RpcRetryingCaller caller = rpcCallerFactory.newCaller();
> try {
> return caller.callWithRetries(callable);
> } finally {
> callable.close();
> }
> }
> {code}
> And inside, the other one is from StubMaker.makeStub():
> {code}
> /**
>* Create a stub against the master. Retry if necessary.
>* @return A stub to do intf against the master
>* @throws MasterNotRunningException
>*/
> @edu.umd.cs.findbugs.annotations.SuppressWarnings
> (value="SWL_SLEEP_WITH_LOCK_HELD")
> Object makeStub() throws MasterNotRunningException {
> {code}
> The tests will just hang for 10 min * 35 ~= 6hours.
> {code}
> 2014-09-23 16:19:05,151 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
> failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,253 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
> failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,456 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
> failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,759 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
> failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:06,262 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
> failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:07,273 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
> failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:09,286 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
> failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:13,303 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
> failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:23,343 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
> failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:33,439 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
> 35 failed; retrying after sleep of 10027, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:43,473 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
> 35 failed; retrying after sleep of 10004, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:53,485 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of
> 35 failed; retrying after sleep of 20160,
[jira] [Updated] (HBASE-11826) Split each tableOrRegionName admin methods into two targetted methods
[
https://issues.apache.org/jira/browse/HBASE-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Sharma updated HBASE-11826:
Fix Version/s: 1.0.0
> Split each tableOrRegionName admin methods into two targetted methods
> -
>
> Key: HBASE-11826
> URL: https://issues.apache.org/jira/browse/HBASE-11826
> Project: HBase
> Issue Type: Improvement
>Reporter: Carter
>Assignee: Carter
> Fix For: 0.99.0, 1.0.0, 2.0.0
>
> Attachments: HBASE_11826.patch, HBASE_11826_v2.patch,
> HBASE_11826_v2.patch, hbase-11826_v3.patch, hbase-11826_v3.patch
>
>
> Purpose of this is two implement [~enis]'s suggestion to strongly type the
> methods that take "tableOrRegionName" as an argument.
> For instance:
> {code}
> void compact(final String tableNameOrRegionName)
> void compact(final byte[] tableNameOrRegionName)
> {code}
> becomes
> {code}
> @Deprecated
> void compact(final String tableNameOrRegionName)
> @Deprecated
> void compact(final byte[] tableNameOrRegionName)
> void compact(TableName table)
> void compactRegion(final byte[] regionName)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12083) Deprecate new HBaseAdmin() in favor of Connection.getAdmin()
[ https://issues.apache.org/jira/browse/HBASE-12083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-12083: Fix Version/s: 1.0.0 > Deprecate new HBaseAdmin() in favor of Connection.getAdmin() > > > Key: HBASE-12083 > URL: https://issues.apache.org/jira/browse/HBASE-12083 > Project: HBase > Issue Type: Bug >Reporter: Solomon Duskis >Assignee: Enis Soztutar >Priority: Critical > Fix For: 1.0.0, 2.0.0, 0.99.2 > > Attachments: hbase-12083_v1.patch, hbase-12083_v2.patch, > hbase-12083_v3-branch-1.patch, hbase-12083_v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14767) Remove deprecated functions from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14767: Attachment: HBASE-14767-master-v2.patch v2 patch: Resolving merge conflicts with patch committed an hour ago. :-/ > Remove deprecated functions from HBaseAdmin > --- > > Key: HBASE-14767 > URL: https://issues.apache.org/jira/browse/HBASE-14767 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14767-master-v2.patch, HBASE-14767-master.patch > > > Many functions in HBaseAdmin are marked deprecated. Removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14769) Remove unused functions and duplicat javadocs from HBaseAdmin
Apekshit Sharma created HBASE-14769: --- Summary: Remove unused functions and duplicat javadocs from HBaseAdmin Key: HBASE-14769 URL: https://issues.apache.org/jira/browse/HBASE-14769 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma HBaseAdmin is marked private, so removing the functions not being used anywhere. Also, the javadocs of overridden functions are same as corresponding ones in Admin.java. Since javadocs are automatically inherited from the interface class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12072) Standardize retry handling for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Sharma updated HBASE-12072:
Fix Version/s: 1.0.0
> Standardize retry handling for master operations
>
>
> Key: HBASE-12072
> URL: https://issues.apache.org/jira/browse/HBASE-12072
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 1.0.0, 2.0.0, 0.99.2
>
> Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
> hbase-12072_v2.patch, hbase-12072_v2.patch, hbase-12072_v3.patch
>
>
> For master requests, there are two retry mechanisms in effect. The first one
> is from HBaseAdmin.executeCallable()
> {code}
> private V executeCallable(MasterCallable callable) throws
> IOException {
> RpcRetryingCaller caller = rpcCallerFactory.newCaller();
> try {
> return caller.callWithRetries(callable);
> } finally {
> callable.close();
> }
> }
> {code}
> And inside, the other one is from StubMaker.makeStub():
> {code}
> /**
>* Create a stub against the master. Retry if necessary.
>* @return A stub to do intf against the master
>* @throws MasterNotRunningException
>*/
> @edu.umd.cs.findbugs.annotations.SuppressWarnings
> (value="SWL_SLEEP_WITH_LOCK_HELD")
> Object makeStub() throws MasterNotRunningException {
> {code}
> The tests will just hang for 10 min * 35 ~= 6hours.
> {code}
> 2014-09-23 16:19:05,151 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
> failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,253 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
> failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,456 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
> failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,759 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
> failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:06,262 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
> failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:07,273 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
> failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:09,286 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
> failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:13,303 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
> failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:23,343 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
> failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:33,439 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
> 35 failed; retrying after sleep of 10027, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:43,473 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
> 35 failed; retrying after sleep of 10004, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:53,485 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of
> 35 failed; retrying after sleep of 20160, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:20:13,656 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 13 of
> 35 failed; retrying after sleep of 20006, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
>
[jira] [Updated] (HBASE-14769) Remove unused functions and duplicat javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14769: Assignee: Apekshit Sharma Status: Patch Available (was: Open) > Remove unused functions and duplicat javadocs from HBaseAdmin > -- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14769) Remove unused functions and duplicat javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14769: Attachment: HBASE-14769-master.patch > Remove unused functions and duplicat javadocs from HBaseAdmin > -- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma > Attachments: HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14767) Remove deprecated functions from HBaseAdmin
Apekshit Sharma created HBASE-14767: --- Summary: Remove deprecated functions from HBaseAdmin Key: HBASE-14767 URL: https://issues.apache.org/jira/browse/HBASE-14767 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Many functions in HBaseAdmin are marked deprecated. Removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14767) Remove deprecated functions from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14767: Status: Patch Available (was: Open) > Remove deprecated functions from HBaseAdmin > --- > > Key: HBASE-14767 > URL: https://issues.apache.org/jira/browse/HBASE-14767 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14767-master.patch > > > Many functions in HBaseAdmin are marked deprecated. Removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14767) Remove deprecated functions from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14767: Attachment: HBASE-14767-master.patch first patch. > Remove deprecated functions from HBaseAdmin > --- > > Key: HBASE-14767 > URL: https://issues.apache.org/jira/browse/HBASE-14767 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma > Attachments: HBASE-14767-master.patch > > > Many functions in HBaseAdmin are marked deprecated. Removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14738) Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988697#comment-14988697 ] Apekshit Sharma commented on HBASE-14738: - In testAllChecksumTypes(), should we also check for the checksum type to make sure it's not using default checksum every time. I see no other test in that file testing the same so might as well do it here. For everything else, LGTM. > Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98 > --- > > Key: HBASE-14738 > URL: https://issues.apache.org/jira/browse/HBASE-14738 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.16 > > Attachments: HBASE-14738-0.98.patch > > > Profiling 0.98.15 I see 20-30% of CPU time spent in Hadoop's PureJavaCrc32. > Not surprising given previous results described on HBASE-11927. Backport. > There are two issues with the backport: > # The patch on 11927 changes the default CRC type from CRC32 to CRC32C. > Although the changes are backwards compatible -files with either CRC type > will be handled correctly in a transparent manner - we should probably leave > the default alone in 0.98 and advise users on a site configuration change to > use CRC32C if desired, for potential hardware acceleration. > # Need a shim for differences between Hadoop's DataChecksum type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14672) Exorcise deprecated Delete#delete* apis
[ https://issues.apache.org/jira/browse/HBASE-14672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979508#comment-14979508 ] Apekshit Sharma commented on HBASE-14672: - lgtm. +1 > Exorcise deprecated Delete#delete* apis > --- > > Key: HBASE-14672 > URL: https://issues.apache.org/jira/browse/HBASE-14672 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14672-v2.patch, hbase-14672-v2.patch, > hbase-14672.patch > > > Delete#delete* apis were replaced with Delete#add* apis. This converts all > instances of it and removes Delete#delete* apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975648#comment-14975648 ] Apekshit Sharma commented on HBASE-14700: - Looking at code, it seems like there exists a way of doing it other way round. I haven't tried it myself. # Client switches to Sasl with [fallbackAllowed|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java#L72]. Till Server has security turned off, it'll prompt client to use simple authentication [here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java#L1527] # Turn on security on server # Turn off fallback on client side. But, there is a big drawback in this method. It can cause service outage if clients aren't configured properly and server side security is turned on. Compared to that, your idea wins big time since it'll allow the clients to migrate one at a time, which gives the ability to control the impact if failure happens. Are there any other ideas you have in mind? > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > #. First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > #. Clients can be converting to using secure authentication incrementally > #. The server audit logs allow identification of clients still using simple > auth to connect > #. Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14675) Excise deprecated Put#add(...) and replace with Put#addColumn(...)
[ https://issues.apache.org/jira/browse/HBASE-14675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968792#comment-14968792 ] Apekshit Sharma commented on HBASE-14675: - +1 Had a quick glance through, just few nits: Params are split across lines and 8 space align instead of 4. RegionMergeTransactionImpl:426 TestFromClientSide:418 Missing comments (in case not removed intentionally) TestFromClientSide:1897 TestRowProcessorEndpoint:138 Alignment: TestHCM:998 - Since it's fixing alignment, the next two lines are misaligned too. TestScannerTimeout:70 - it's 'for' that needs to be aligned TestFilter:1819 - new lines are misaligned. TestCellCounter:124 & 164 & 204 - The block will look funny due staggered alignment of rows. I believe there are few lines which exceed 100 char limit, but pre-commit test should notify those. > Excise deprecated Put#add(...) and replace with Put#addColumn(...) > -- > > Key: HBASE-14675 > URL: https://issues.apache.org/jira/browse/HBASE-14675 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14675.patch > > > The Put API changed from #add(...) to #addColumn(...). This updates all > instances of it and removes it from the Put (which was added for hbase 1.0.0) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14666) Remove deprecated HBaseTestingUtility#deleteTable methods
[ https://issues.apache.org/jira/browse/HBASE-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968524#comment-14968524 ] Apekshit Sharma commented on HBASE-14666: - +1 > Remove deprecated HBaseTestingUtility#deleteTable methods > - > > Key: HBASE-14666 > URL: https://issues.apache.org/jira/browse/HBASE-14666 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14666.patch > > > HBase has had the TableName APIs since 0.96 and our test code doesn't have > privacy/stability markers so we should be able to remove the deprecated > methods in 2.0.0. > deleteTable has some separate unit test related cleanup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14665) Remove deprecated HBaseTestingUtility#createTable methods
[ https://issues.apache.org/jira/browse/HBASE-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968506#comment-14968506 ] Apekshit Sharma commented on HBASE-14665: - The methods being removed here were marked deprecated in HBASE-13893 and will be released in 2.0.0. Since the class is marked public and evolving, we should keep the methods around at least till 2.1 to meet the stability guidelines. However I personally feel that given we moved to TableName api in 0.96, we should be fine to remove it, and in fact should do it. > Remove deprecated HBaseTestingUtility#createTable methods > - > > Key: HBASE-14665 > URL: https://issues.apache.org/jira/browse/HBASE-14665 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14665.patch > > > HBase has had the TableName APIs since 0.96 and our test code doesn't have > privacy/stability markers so we should be able to remove the deprecated > methods in 2.0.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14469) Fix some comment, validation and logging around memstore lower limit configuration
[ https://issues.apache.org/jira/browse/HBASE-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907652#comment-14907652 ] Apekshit Sharma commented on HBASE-14469: - Thanks [~anoop.hbase] for review and [~mbertozzi] for review and committing. :-) > Fix some comment, validation and logging around memstore lower limit > configuration > -- > > Key: HBASE-14469 > URL: https://issues.apache.org/jira/browse/HBASE-14469 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-14469-master.patch > > > Trivial code cleanup around memstore lower limit configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14469) Fix some comment, validation and logging around memstore lower limit configuration
[ https://issues.apache.org/jira/browse/HBASE-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906874#comment-14906874 ] Apekshit Sharma commented on HBASE-14469: - Ready to go in? > Fix some comment, validation and logging around memstore lower limit > configuration > -- > > Key: HBASE-14469 > URL: https://issues.apache.org/jira/browse/HBASE-14469 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Minor > Attachments: HBASE-14469-master.patch > > > Trivial code cleanup around memstore lower limit configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14469) Fix some comment, validation and logging around memstore lower limit configuration
[ https://issues.apache.org/jira/browse/HBASE-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14469: Status: Patch Available (was: Open) > Fix some comment, validation and logging around memstore lower limit > configuration > -- > > Key: HBASE-14469 > URL: https://issues.apache.org/jira/browse/HBASE-14469 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Minor > Attachments: HBASE-14469-master.patch > > > Trivial code cleanup around memstore lower limit configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14469) Fix some comment, validation and logging around memstore lower limit configuration
[ https://issues.apache.org/jira/browse/HBASE-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14469: Attachment: HBASE-14469-master.patch > Fix some comment, validation and logging around memstore lower limit > configuration > -- > > Key: HBASE-14469 > URL: https://issues.apache.org/jira/browse/HBASE-14469 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Minor > Attachments: HBASE-14469-master.patch > > > Trivial code cleanup around memstore lower limit configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14469) Fix some comment, validation and logging around memstore lower limit configuration
Apekshit Sharma created HBASE-14469: --- Summary: Fix some comment, validation and logging around memstore lower limit configuration Key: HBASE-14469 URL: https://issues.apache.org/jira/browse/HBASE-14469 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Priority: Minor Trivial code cleanup around memstore lower limit configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743860#comment-14743860 ] Apekshit Sharma commented on HBASE-14400: - [~apurtell] [~mbertozzi] is there anything needed from my side that I might be missing? > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-0.98.patch, > HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch, > HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch, > HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Release Note: To use rpc protection in HBase, set the value of 'hbase.rpc.protection' to: 'authentication' : simple authentication using kerberos 'integrity' : authentication and integrity 'privacy' : authentication and confidentiality Earlier, HBase reference guide erroneously mentioned in some places to set the value to 'auth-conf'. This patch fixes the guide and adds temporary support for erroneously recommended values. > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-1.0.patch, > HBASE-14400-branch-1.1.patch, HBASE-14400-branch-1.2.patch, > HBASE-14400-master-v2.patch, HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741364#comment-14741364 ] Apekshit Sharma commented on HBASE-14400: - *attached patch for it now. > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-0.98.patch, > HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch, > HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch, > HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741363#comment-14741363 ] Apekshit Sharma commented on HBASE-14400: - Yes. attached patch for it. > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-0.98.patch, > HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch, > HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch, > HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Attachment: HBASE-14400-branch-0.98.patch > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-0.98.patch, > HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch, > HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch, > HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Attachment: HBASE-14400-branch-1.2.patch HBASE-14400-branch-1.1.patch HBASE-14400-branch-1.0.patch > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-14400-branch-1.0.patch, > HBASE-14400-branch-1.1.patch, HBASE-14400-branch-1.2.patch, > HBASE-14400-master-v2.patch, HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Fix Version/s: (was: 1.3.0) 1.1.3 1.0.3 > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-1.0.patch, > HBASE-14400-branch-1.1.patch, HBASE-14400-branch-1.2.patch, > HBASE-14400-master-v2.patch, HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Attachment: HBASE-14400-master-v2.patch > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-14400-master-v2.patch, HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739854#comment-14739854 ] Apekshit Sharma commented on HBASE-14400: - We want to warn the users if they use auth/auth-int/auth-conf as value, instead of using authentication/integrity/privacy. Here QualityOfProtection.INTEGRITY.saslQop is 'auth-int' and similarly others. So it's fine. > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-14400-master-v2.patch, HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Fix Version/s: 1.1.3 1.0.3 1.2.1 1.3.0 2.0.0 > Fix HBase RPC protection > > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3 > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Attachment: HBASE-14400-master.patch > Fix HBase RPC protection > > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Fix Version/s: (was: 1.1.3) (was: 1.0.3) > Fix HBase RPC protection > > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14400) Fix HBase RPC protection
Apekshit Sharma created HBASE-14400: --- Summary: Fix HBase RPC protection Key: HBASE-14400 URL: https://issues.apache.org/jira/browse/HBASE-14400 Project: HBase Issue Type: Bug Components: encryption, rpc, security Reporter: Apekshit Sharma Assignee: Apekshit Sharma Priority: Critical HBase configuration 'hbase.rpc.protection' can be set to 'authentication', 'integrity' or 'privacy'. "authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled." However hbase ref guide incorrectly suggests in some places to set the value to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't provide rpc encryption which is what user wants. This jira will fix: - documentation: change 'auth-conf' references to 'privacy' - SaslUtil to support both set of values (privacy/integrity/authentication and auth-conf/auth-int/auth) to be backward compatible with what was being suggested till now. - change 'hbase.thrift.security.qop' to be consistent with other similar configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7105) RS throws NPE on forcing compaction from HBase shell on a single bulk imported file.
[
https://issues.apache.org/jira/browse/HBASE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654602#comment-14654602
]
Apekshit Sharma commented on HBASE-7105:
I am herewaiting for some action.
I believe the current patch has correctness issues, but the system as of now
can be improved.
How costly is it to open the reader?
If not a lot, then maybe few extra IOs are less trouble then exceptions from UX
point of view.
RS throws NPE on forcing compaction from HBase shell on a single bulk
imported file.
Key: HBASE-7105
URL: https://issues.apache.org/jira/browse/HBASE-7105
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Karthik Ranganathan
Assignee: Cosmin Lehene
Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3
Attachments:
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch,
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch
In StoreFile, we have:
private AtomicBoolean majorCompaction = null;
In StoreFile.open(), we do:
b = metadataMap.get(MAJOR_COMPACTION_KEY);
if (b != null) {
// init majorCompaction variable
}
Because the file was bulk imported, this is never initialized. Any subsequent
call to isMajorCompaction() NPE's.
Fix is to initialize it to false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650503#comment-14650503 ] Apekshit Sharma commented on HBASE-14148: - [~busbey] Should be cool to use the test from earlier patch because it was purely my contribution. I don't believe we need to cleanroom it. [~gliptak] If sean agrees, feel free to copy the tests. Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Gabor Liptak Attachments: HBASE-14148-cleanroom.1.patch, HBASE-14148-master.patch, HBASE-14148-v2-master.patch, HBASE-14148-v3-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642332#comment-14642332 ] Apekshit Sharma commented on HBASE-14148: - I have looked at that code again and again but I don't think i'll be able to change anything in it, mainly because there is no meat in there. The whole episode just makes me sad, but I understand. :( What to do next? [~busbey] Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-14148-master.patch, HBASE-14148-v2-master.patch, HBASE-14148-v3-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14148: Attachment: HBASE-14148-v3-master.patch Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-14148-master.patch, HBASE-14148-v2-master.patch, HBASE-14148-v3-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14148) Web UI Framable Page
[
https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641279#comment-14641279
]
Apekshit Sharma commented on HBASE-14148:
-
In v2, license in the start of XFrameOptionsFilter.java was
{noformat}
/**
* Software published by the Open Web Application Security Project
(http://www.owasp.org)
* This software is licensed under the new BSD license.
* Copyright 2009, Jeff Williams a
href=http://www.aspectsecurity.com;Aspect Security/a
*/
{noformat}
I thought -1 release audit was because of that, so I changed it to full
license in V3. Alas, QA is still cribbing. What should I do [~busbey]? Or is it
supposed to be like that?
Web UI Framable Page
Key: HBASE-14148
URL: https://issues.apache.org/jira/browse/HBASE-14148
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Attachments: HBASE-14148-master.patch, HBASE-14148-v2-master.patch,
HBASE-14148-v3-master.patch
The web UIs do not include the X-Frame-Options header to prevent the pages
from being framed from another site.
Reference:
https://www.owasp.org/index.php/Clickjacking
https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet
https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14148: Attachment: HBASE-14148-v2-master.patch Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-14148-master.patch, HBASE-14148-v2-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14639408#comment-14639408 ] Apekshit Sharma commented on HBASE-14148: - [~eclark] bq. I don't think that this is complex enough that we should be copying code from others. I agree with you that it is very trivial piece of code and that anyone would have written same. However, when i was reading on clickjacking, i came across it, read it and it was in the back of my head when I was doing changes. So the fact remains, that i referenced it and since the code looks same (it couldn't have looked different, right!), I'd would prefer to have the new-BSD license here. We all are anyways interested in the feature, right? Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-14148-master.patch, HBASE-14148-v2-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14148) Web UI Framable Page
Apekshit Sharma created HBASE-14148: --- Summary: Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14148: Status: Patch Available (was: Open) Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-14148-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14148: Attachment: HBASE-14148-master.patch Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-14148-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14127) Refactor AssignmentManager.unassign() function which has lots of return statements
[ https://issues.apache.org/jira/browse/HBASE-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14127: Attachment: HBASE-14127-master.patch should be fine to do this refactoring only in master branch since branch-1 has zk based assignment manager and is already significantly different. Refactor AssignmentManager.unassign() function which has lots of return statements -- Key: HBASE-14127 URL: https://issues.apache.org/jira/browse/HBASE-14127 Project: HBase Issue Type: Improvement Reporter: Apekshit Sharma Assignee: Apekshit Sharma Priority: Minor Attachments: HBASE-14127-master.patch In AssignmentManager.unassign() * there are lots of return points * if condition is very non-trivial to understand * reuse of 'state' for different purpose makes things confusing A simple unfold of if-else with with better variable names should be sufficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14127) Refactor AssignmentManager.unassign() function which has lots of return statements
Apekshit Sharma created HBASE-14127: --- Summary: Refactor AssignmentManager.unassign() function which has lots of return statements Key: HBASE-14127 URL: https://issues.apache.org/jira/browse/HBASE-14127 Project: HBase Issue Type: Improvement Reporter: Apekshit Sharma Assignee: Apekshit Sharma Priority: Minor In AssignmentManager.unassign() * there are lots of return points * if condition is very non-trivial to understand * reuse of 'state' for different purpose makes things confusing A simple unfold of if-else with with better variable names should be sufficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14119) Show meaningful error messages instead of stack traces in hbase shell commands. Fixing few commands in this jira.
[
https://issues.apache.org/jira/browse/HBASE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Sharma updated HBASE-14119:
Attachment: HBASE-14119-branch-1.patch
branch-1 patch
Show meaningful error messages instead of stack traces in hbase shell
commands. Fixing few commands in this jira.
-
Key: HBASE-14119
URL: https://issues.apache.org/jira/browse/HBASE-14119
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Priority: Minor
Attachments: HBASE-14119-branch-1.patch, HBASE-14119.patch
This isn't really a functional bug, just more about erroring out cleanly.
In the future, everyone should check and catch exceptions. Meaningful error
messages should be shown instead of stack traces. For debugging purposes,
'hbase shell -d' can be used which outputs a detailed stack trace.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException:
Snapshot 'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not
leave stacktrace in the output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14119) Show error message instead of stack traces in hbase shell commands
[
https://issues.apache.org/jira/browse/HBASE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Sharma updated HBASE-14119:
Status: Patch Available (was: In Progress)
Show error message instead of stack traces in hbase shell commands
--
Key: HBASE-14119
URL: https://issues.apache.org/jira/browse/HBASE-14119
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Priority: Minor
Attachments: HBASE-14119.patch
This isn't really a functional bug, just more about erroring out cleanly.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException:
Snapshot 'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not
leave stacktrace in the output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14119) Show meaningful error messages instead of stack traces in hbase shell commands. Fixing few commands in this jira.
[
https://issues.apache.org/jira/browse/HBASE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Sharma updated HBASE-14119:
Description:
This isn't really a functional bug, just more about erroring out cleanly.
In the future, everyone should check and catch exceptions. Meaningful error
messages should be shown instead of stack traces. For debugging purposes,
'hbase shell -d' can be used which outputs a detailed stack trace.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot
'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not leave
stacktrace in the output.
was:
This isn't really a functional bug, just more about erroring out cleanly.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
[jira] [Updated] (HBASE-14119) Show meaningful error messages instead of stack traces in hbase shell commands. Fixing few commands in this jira.
[
https://issues.apache.org/jira/browse/HBASE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Sharma updated HBASE-14119:
Summary: Show meaningful error messages instead of stack traces in hbase
shell commands. Fixing few commands in this jira. (was: Show error message
instead of stack traces in hbase shell commands)
Show meaningful error messages instead of stack traces in hbase shell
commands. Fixing few commands in this jira.
-
Key: HBASE-14119
URL: https://issues.apache.org/jira/browse/HBASE-14119
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Priority: Minor
Attachments: HBASE-14119.patch
This isn't really a functional bug, just more about erroring out cleanly.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException:
Snapshot 'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not
leave stacktrace in the output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14119) Show error message instead of stack traces in hbase shell commands
[
https://issues.apache.org/jira/browse/HBASE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632221#comment-14632221
]
Apekshit Sharma commented on HBASE-14119:
-
patch attached.
Show error message instead of stack traces in hbase shell commands
--
Key: HBASE-14119
URL: https://issues.apache.org/jira/browse/HBASE-14119
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Priority: Minor
Attachments: HBASE-14119.patch
This isn't really a functional bug, just more about erroring out cleanly.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException:
Snapshot 'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not
leave stacktrace in the output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14119) Show error message instead of stack traces in hbase shell commands
[
https://issues.apache.org/jira/browse/HBASE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Sharma updated HBASE-14119:
Attachment: HBASE-14119.patch
Show error message instead of stack traces in hbase shell commands
--
Key: HBASE-14119
URL: https://issues.apache.org/jira/browse/HBASE-14119
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Priority: Minor
Attachments: HBASE-14119.patch
This isn't really a functional bug, just more about erroring out cleanly.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException:
Snapshot 'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not
leave stacktrace in the output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-14119) Show error message instead of stack traces in hbase shell commands
[
https://issues.apache.org/jira/browse/HBASE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-14119 started by Apekshit Sharma.
---
Show error message instead of stack traces in hbase shell commands
--
Key: HBASE-14119
URL: https://issues.apache.org/jira/browse/HBASE-14119
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Priority: Minor
Attachments: HBASE-14119.patch
This isn't really a functional bug, just more about erroring out cleanly.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException:
Snapshot 'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not
leave stacktrace in the output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-7105) RS throws NPE on forcing compaction from HBase shell on a single bulk imported file.
[
https://issues.apache.org/jira/browse/HBASE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632239#comment-14632239
]
Apekshit Sharma commented on HBASE-7105:
[~enis] [~clehene]
I also don't think that this is specifically related to bulk files in any way.
Looks like the main issue here is that 'majorCompacted' field's value requires
the reader to be open and is invalid until then, however the functions to get
the value and open the reader does not reflect that dependence in relationship
and can be called independently. This has led to code like
{code}
HStore.java
@Override
public boolean isMajorCompaction() throws IOException {
for (StoreFile sf : this.storeEngine.getStoreFileManager().getStorefiles())
{
// TODO: what are these reader checks all over the place?
if (sf.getReader() == null) {
LOG.debug(StoreFile + sf + has null Reader);
return false;
}
}
return storeEngine.getCompactionPolicy().isMajorCompaction(
this.storeEngine.getStoreFileManager().getStorefiles());
}
{code}
The fix suggested in description seems like a bad thing to do because if
createReader() isn't called first for whatever reason, it may incorrectly
advertise a store file as non-major when it in fact was a major compaction
file. At least current code fails correctly.
Alternatives:
* open the reader explicitly if this.reader == null when isMajorCompacted() is
called. However, this will make subsequent createReader() call to throw
IllegalAccessError. (my choice would be keeping existing code than doing this)
* move the function to Reader class. (i feel like this will be best thing to do)
RS throws NPE on forcing compaction from HBase shell on a single bulk
imported file.
Key: HBASE-7105
URL: https://issues.apache.org/jira/browse/HBASE-7105
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Karthik Ranganathan
Assignee: Cosmin Lehene
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments:
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch,
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch
In StoreFile, we have:
private AtomicBoolean majorCompaction = null;
In StoreFile.open(), we do:
b = metadataMap.get(MAJOR_COMPACTION_KEY);
if (b != null) {
// init majorCompaction variable
}
Because the file was bulk imported, this is never initialized. Any subsequent
call to isMajorCompaction() NPE's.
Fix is to initialize it to false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14119) Show error message instead of stack traces in hbase shell commands
Apekshit Sharma created HBASE-14119:
---
Summary: Show error message instead of stack traces in hbase shell
commands
Key: HBASE-14119
URL: https://issues.apache.org/jira/browse/HBASE-14119
Project: HBase
Issue Type: Bug
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
Priority: Minor
This isn't really a functional bug, just more about erroring out cleanly.
* the shell commands assign, move, unassign and merge_region can throw the
following error if given an invalid argument:
{noformat}
hbase(main):032:0 unassign 'adsfdsafdsa'
ERROR: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.UnknownRegionException: adsfdsafdsa
at org.apache.hadoop.hbase.master.HMaster.unassign(HMaster.java:1562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Here is some help for this command:
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase unassign 'REGIONNAME'
hbase unassign 'REGIONNAME', true
hbase(main):033:0
{noformat}
* drop_namespace, describe_namespace throw stack trace too.
{noformat}
hbase(main):002:0 drop_namespace SDf
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: SDf
at
org.apache.hadoop.hbase.master.TableNamespaceManager.remove(TableNamespaceManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2119)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:430)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44279)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Drop the named namespace. The namespace must be empty.
{noformat}
* fix error message in close_region
{noformat}
hbase(main):007:0 close_region sdf
ERROR: sdf
{noformat}
* delete_snapshot throws exception too.
{noformat}
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot
'sdf' doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:270)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:452)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44261)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Delete a specified snapshot. Examples:
hbase delete_snapshot 'snapshotName',
{noformat}
other commands, when given bogus arguments, tend to fail cleanly and not leave
stacktrace in the output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14083) Fix separator width in Backup Masters of WebUI
[ https://issues.apache.org/jira/browse/HBASE-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628394#comment-14628394 ] Apekshit Sharma commented on HBASE-14083: - +1. Fix separator width in Backup Masters of WebUI -- Key: HBASE-14083 URL: https://issues.apache.org/jira/browse/HBASE-14083 Project: HBase Issue Type: Bug Components: UI Reporter: Yuhao Bi Assignee: Yuhao Bi Priority: Minor Attachments: After.PNG, Before.PNG, HBASE-14083.patch The horizontal line separator above total count in backup master is shorter than the others. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627391#comment-14627391
]
Apekshit Sharma commented on HBASE-13981:
-
Great, comments are fixed and existing wrong configuration has been deprecated.
However, a new configuration that'll eventually replace the deprecated should
be added too.
(from last comment)
{code}
+ @Deprecated
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
+ public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.separator;
{code}
[~larsgeorge] this patch should be ready soon. Can you please commit it when
it's ready. (i don't have permissions)
Fix ImportTsv spelling and usage issues
---
Key: HBASE-13981
URL: https://issues.apache.org/jira/browse/HBASE-13981
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Gabor Liptak
Labels: beginner
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-13981.1.patch, HBASE-13981.2.patch,
HBASE-13981.3.patch, HBASE-13981.4.patch
The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
In code:
{noformat}
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
{noformat}
It is separator.
In usage text:
{noformat}
input data. Another special columnHBASE_TS_KEY designates that this column
should be
{noformat}
Space missing.
{noformat}
Record with invalid timestamps (blank, non-numeric) will be treated as bad
record.
{noformat}
Records ... as bad records - plural missing twice.
{noformat}
HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
Should be specified as key=value where -1 is used
as the seperator. Note that more than one OperationAttributes can be
specified.
{noformat}
- Remove line wraps and indentation.
- Fix separator.
- Fix wrong separator being output, it is not -1 (wrong constant use in
code)
- General wording/style could be better (eg. last sentence now uses
OperationAttributes without a space).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627454#comment-14627454
]
Apekshit Sharma commented on HBASE-13981:
-
[~gliptak] Yeah, you're right.
+1.
Fix ImportTsv spelling and usage issues
---
Key: HBASE-13981
URL: https://issues.apache.org/jira/browse/HBASE-13981
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Gabor Liptak
Labels: beginner
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-13981.1.patch, HBASE-13981.2.patch,
HBASE-13981.3.patch, HBASE-13981.4.patch
The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
In code:
{noformat}
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
{noformat}
It is separator.
In usage text:
{noformat}
input data. Another special columnHBASE_TS_KEY designates that this column
should be
{noformat}
Space missing.
{noformat}
Record with invalid timestamps (blank, non-numeric) will be treated as bad
record.
{noformat}
Records ... as bad records - plural missing twice.
{noformat}
HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
Should be specified as key=value where -1 is used
as the seperator. Note that more than one OperationAttributes can be
specified.
{noformat}
- Remove line wraps and indentation.
- Fix separator.
- Fix wrong separator being output, it is not -1 (wrong constant use in
code)
- General wording/style could be better (eg. last sentence now uses
OperationAttributes without a space).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615940#comment-14615940
]
Apekshit Sharma commented on HBASE-13981:
-
[~gliptak] not sure why ATTRIBUTE_SEPERATOR_CONF_KEY or attributes.seperator
are there. Since ImportTsv is @InterfaceAudience.Public, we can not simply
delete it but can mark it @Deprecated.
Patch looks good. Just last two things.
1.
{code}
- public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
+ public final static String ATTRIBUTE_SEPARATOR_CONF_KEY =
attributes.separator;
...
- final static String DEFAULT_ATTRIBUTES_SEPERATOR = =;
- final static String DEFAULT_MULTIPLE_ATTRIBUTES_SEPERATOR = ,;
+ final static String DEFAULT_ATTRIBUTES_SEPARATOR = =;
+ final static String DEFAULT_MULTIPLE_ATTRIBUTES_SEPARATOR = ,;
{code}
Again, since ImportTsv is @InterfaceAudience.Public (read more about this
[here|http://hbase.apache.org/book.html#hbase.client.api.surface]), we can not
simply change the name. The right thing to do here would be
{code}
+ @Deprecated
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
+ public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.separator;
{code}
and replacing all uses of ATTRIBUTE_SEPERATOR_CONF_KEY with
ATTRIBUTE_SEPERATOR_CONF_KEY. Later in 2.0, the deprecated will be deleted.
2. Readability: there is no need for indentation here.
{code}
+ The column names of the TSV data must be specified using the option:\n
+
+-D + COLUMNS_CONF_KEY + option. This option takes the form
of +
+ comma-separated column names, where each column name is either
+
{code}
Please align like this
{code}
+ The column names of the TSV data must be specified using the option:\n
+
+-D + COLUMNS_CONF_KEY + option. This option takes the form of +
+ comma-separated column names, where each column name is either +
{code}
here and everywhere else.
Fix ImportTsv spelling and usage issues
---
Key: HBASE-13981
URL: https://issues.apache.org/jira/browse/HBASE-13981
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Gabor Liptak
Labels: beginner
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-13981.1.patch, HBASE-13981.2.patch,
HBASE-13981.3.patch
The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
In code:
{noformat}
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
{noformat}
It is separator.
In usage text:
{noformat}
input data. Another special columnHBASE_TS_KEY designates that this column
should be
{noformat}
Space missing.
{noformat}
Record with invalid timestamps (blank, non-numeric) will be treated as bad
record.
{noformat}
Records ... as bad records - plural missing twice.
{noformat}
HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
Should be specified as key=value where -1 is used
as the seperator. Note that more than one OperationAttributes can be
specified.
{noformat}
- Remove line wraps and indentation.
- Fix separator.
- Fix wrong separator being output, it is not -1 (wrong constant use in
code)
- General wording/style could be better (eg. last sentence now uses
OperationAttributes without a space).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-branch-1-v3.patch [~tedyu] fixed the test. So one of the test was failing because an existing test was directly changing global configuration (util.getConfiguration()) in its test body which affected any tests that ran later. :-/ ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Fix For: 2.0.0, 1.3.0 Attachments: HBASE-13702-branch-1-v2.patch, HBASE-13702-branch-1-v3.patch, HBASE-13702-branch-1.patch, HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702-v4.patch, HBASE-13702-v5.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14009) Correct spelling for ImportTsv ATTRIBUTE_SEPERATOR_CONF_KEY
[ https://issues.apache.org/jira/browse/HBASE-14009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14611398#comment-14611398 ] Apekshit Sharma commented on HBASE-14009: - I agree with Anoop since it will be just additional 5 lines or so. Correct spelling for ImportTsv ATTRIBUTE_SEPERATOR_CONF_KEY --- Key: HBASE-14009 URL: https://issues.apache.org/jira/browse/HBASE-14009 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Priority: Minor Labels: breaking-api -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14611407#comment-14611407
]
Apekshit Sharma commented on HBASE-13981:
-
Thanks for doing additional changes style changes.
just 2 minor things.
extra quote sign after =|
{code}
-D + SEPARATOR_CONF_KEY + =|' - eg separate on pipes instead of tabs\n +
{code}
mis-alignment here and in other similar places. Should be aligned with previous
line.
{code}
The column names of the TSV data must be specified using the -D +
COLUMNS_CONF_KEY + option. This option takes the form of +
{code}
Fix ImportTsv spelling and usage issues
---
Key: HBASE-13981
URL: https://issues.apache.org/jira/browse/HBASE-13981
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Gabor Liptak
Labels: beginner
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-13981.1.patch, HBASE-13981.2.patch
The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
In code:
{noformat}
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
{noformat}
It is separator.
In usage text:
{noformat}
input data. Another special columnHBASE_TS_KEY designates that this column
should be
{noformat}
Space missing.
{noformat}
Record with invalid timestamps (blank, non-numeric) will be treated as bad
record.
{noformat}
Records ... as bad records - plural missing twice.
{noformat}
HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
Should be specified as key=value where -1 is used
as the seperator. Note that more than one OperationAttributes can be
specified.
{noformat}
- Remove line wraps and indentation.
- Fix separator.
- Fix wrong separator being output, it is not -1 (wrong constant use in
code)
- General wording/style could be better (eg. last sentence now uses
OperationAttributes without a space).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-branch-1.patch [~tedyu] Sorry for the late response. PFA patch for branch-1. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Fix For: 2.0.0, 1.3.0 Attachments: HBASE-13702-branch-1.patch, HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702-v4.patch, HBASE-13702-v5.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13957) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13957: Attachment: HBASE-13957-v2.patch Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13957-v2.patch, HBASE-13957.patch This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new class ConfigurationProperty to get a better sense of use patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-branch-1-v2.patch ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Fix For: 2.0.0, 1.3.0 Attachments: HBASE-13702-branch-1-v2.patch, HBASE-13702-branch-1.patch, HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702-v4.patch, HBASE-13702-v5.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma reassigned HBASE-13936: --- Assignee: Apekshit Sharma Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Umbrella Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: DynamicConfigs.v01.docx, design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13981) Fix ImportTsv spelling and usage issues
[
https://issues.apache.org/jira/browse/HBASE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606060#comment-14606060
]
Apekshit Sharma commented on HBASE-13981:
-
While you are at it, maybe fix that whole block of code because there are lots
of style issues in there. One or two are easy to ignore, but so many are just
eyesore! It's fine if you can't right now, in which case feel free to create a
new jira and assign it to me.
1) Weird new line after 'be'
{code}
input data. Another special column + TsvParser.TIMESTAMPKEY_COLUMN_SPEC +
designates that this column should be\n +
used as timestamp for each record. Unlike + TsvParser.ROWKEY_COLUMN_SPEC +
, +
{code}
In fact, it feels like '\n' has been overused by added to the end of each line.
Probably only use it when paragraph break is required?
2) Inconsistent spaces: Sometimes there are spaces around '+' and sometimes
not. Adding spaces everywhere would be good idea.
3) wrong alignment
{code}
Attributes should be specified as
key+ImportTsv.DEFAULT_ATTRIBUTES_SEPARATOR+value where +
ImportTsv.DEFAULT_MULTIPLE_ATTRIBUTES_SEPARATOR+
is used as the separator. Note that more than one Operation Attributes can be
specified.\n+
{code}
Fix ImportTsv spelling and usage issues
---
Key: HBASE-13981
URL: https://issues.apache.org/jira/browse/HBASE-13981
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Gabor Liptak
Labels: beginner
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13981.1.patch
The {{ImportTsv}} tool has various spelling and formatting issues. Fix those.
In code:
{noformat}
public final static String ATTRIBUTE_SEPERATOR_CONF_KEY =
attributes.seperator;
{noformat}
It is separator.
In usage text:
{noformat}
input data. Another special columnHBASE_TS_KEY designates that this column
should be
{noformat}
Space missing.
{noformat}
Record with invalid timestamps (blank, non-numeric) will be treated as bad
record.
{noformat}
Records ... as bad records - plural missing twice.
{noformat}
HBASE_ATTRIBUTES_KEY can be used to specify Operation Attributes per record.
Should be specified as key=value where -1 is used
as the seperator. Note that more than one OperationAttributes can be
specified.
{noformat}
- Remove line wraps and indentation.
- Fix separator.
- Fix wrong separator being output, it is not -1 (wrong constant use in
code)
- General wording/style could be better (eg. last sentence now uses
OperationAttributes without a space).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-13957) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma reassigned HBASE-13957: --- Assignee: Apekshit Sharma Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13957.patch This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new class ConfigurationProperty to get a better sense of use patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13957) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13957: Description: This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new class ConfigurationProperty to get a better sense of use patterns. (was: This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new enum HConfig to get a better sense of use patterns.) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma Attachments: HBASE-13957.patch This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new class ConfigurationProperty to get a better sense of use patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13957) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13957: Summary: Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty (was: Add enum HConfig. Add new functions to ConfigurationManager which use HConfig.) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma Attachments: HBASE-13957.patch This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new enum HConfig to get a better sense of use patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13957) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605027#comment-14605027 ] Apekshit Sharma commented on HBASE-13957: - new changes are up on review board now. (discarded old board because of major changes in design) Add ConfigurationProperty. Add new functions to ConfigurationManager which use the new ConfigurationProperty Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma Attachments: HBASE-13957.patch This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new class ConfigurationProperty to get a better sense of use patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604404#comment-14604404 ] Apekshit Sharma commented on HBASE-13936: - I think that Hadoop style site files is an good design and it should be left as such. The scope of this project and the design changes we have thought so far (in the doc) will be invisible to the users and will only impact dev. bq. Moving from Configuration to ConfigurationManager So the basic idea is to encapsulate Configuration within ConfigurationManager and provide a better API for handling configurations. That will help in building a better framework for dynamic configurations, type check configuration values, and get rid of few other bad patterns. Since the aim here is to promote right patterns (and possibly design the framework so that it's not possible to go otherwise), I will highlight major issues here and get everyone's opinions. [~apurtell] On that note, what do you about the issue of set*() functions (my last post). Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Umbrella Reporter: Apekshit Sharma Attachments: DynamicConfigs.v01.docx, design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603577#comment-14603577 ] Apekshit Sharma commented on HBASE-13936: - Thanks [~enis]. Comments inlined. bq. Prefixing 'H' is deprecated. Please refrain from that old pattern. Will keep in mind. bq. Showing every possible configuration value to a single class is an anti-pattern. HConstants was that, and it went bad. The configurations should be defined close to where they are used. That's right. I didn't realize that until [~mbertozzi] explained it to me yesterday. We have decided to change the design in favor of grouping configurations at client, server, common, etc level. What do you think? bq. Please take a look at TableConfiguration, and why it was introduced. In a lot of places, the configuration parsing becomes the bottleneck especially in the hot code paths. Going forward we should only be passing small parsed POJO style objects holding the configuration for that specific sub system. A bad example is CacheConfig. It holds the defaults, config names, and actual values together with the actual Block cache reference. But the idea should be keeping the parsing / deprecation handling / business logic and the actual parsed values related to a single system together and use that object to pass it around. I agree with you, I have seen and used this design in other places, and I like it most because it is very clean. However, do you think there is something we can change in the design of configuration framework i.e. Configuration , ConfigurationManager and other core classes (i'll be adding some new classes, will add you on review board), to promote or push this design? Because it seems to me that it's more about educating the community. Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Umbrella Reporter: Apekshit Sharma Attachments: DynamicConfigs.v01.docx, design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603586#comment-14603586 ] Apekshit Sharma commented on HBASE-13936: - bq. Please include hbase group in review request Definitely, isn't that default? :) Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Umbrella Reporter: Apekshit Sharma Attachments: DynamicConfigs.v01.docx, design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603641#comment-14603641 ] Apekshit Sharma commented on HBASE-13936: - While we are at it, I'd like to throw some other big questions/doubts/ideas here. Starting with this one, I feel like having set*() function for configurations isn't a good idea. It essentially makes the Configuration object a big container of 100s of variables being changed and passed around. In our code, 'conf' is copied around, values are overwritten, then passed around again. Scary. Makes it quite easy to screw up. I believe that a better design would be having a single immutable Configuration object. Any case which requires overwriting a configuration property inside the code should be refactor-able into a better design. What do others think? Is there any case where overwriting a property inside code is the only or the best way? (unit-tests aside) Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Umbrella Reporter: Apekshit Sharma Attachments: DynamicConfigs.v01.docx, design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-v5.patch [~tedyu] you're right. fixed the issues. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702-v4.patch, HBASE-13702-v5.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Status: Patch Available (was: Open) ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702-v4.patch, HBASE-13702-v5.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13957) Add enum HConfig. Add new functions to ConfigurationManager which use HConfig.
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13957: Issue Type: Sub-task (was: Task) Parent: HBASE-13936 Add enum HConfig. Add new functions to ConfigurationManager which use HConfig. -- Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13957) Add enum HConfig. Add new functions to ConfigurationManager which use HConfig.
Apekshit Sharma created HBASE-13957: --- Summary: Add enum HConfig. Add new functions to ConfigurationManager which use HConfig. Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Task Reporter: Apekshit Sharma -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13957) Add enum HConfig. Add new functions to ConfigurationManager which use HConfig.
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13957: Attachment: HBASE-13957.patch Add enum HConfig. Add new functions to ConfigurationManager which use HConfig. -- Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma Attachments: HBASE-13957.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13957) Add enum HConfig. Add new functions to ConfigurationManager which use HConfig.
[ https://issues.apache.org/jira/browse/HBASE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13957: Description: This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new enum HConfig to get a better sense of use patterns. Add enum HConfig. Add new functions to ConfigurationManager which use HConfig. -- Key: HBASE-13957 URL: https://issues.apache.org/jira/browse/HBASE-13957 Project: HBase Issue Type: Sub-task Reporter: Apekshit Sharma Attachments: HBASE-13957.patch This sub-task is for adding core functionality for configuration improvements. Few configurations will be added/moved to new enum HConfig to get a better sense of use patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-v2.patch v2 patch. fixed some naming issues and merge conflicts. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702-v2.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-v3.patch ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-v4.patch ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702-v4.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595416#comment-14595416 ] Apekshit Sharma commented on HBASE-13702: - Hey guys, I'd really like to close this one since it's almost there. Requesting reviews. Thanks. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13936) Improve dynamic configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13936: Description: Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! was: Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.jpg! Improve dynamic configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Attachments: design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13936) Improve dynamic configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13936: Attachment: design.png Improve dynamic configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Attachments: design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.jpg! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13936) Improve dynamic configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13936: Description: Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.jpg! was:Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Improve dynamic configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.jpg! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13936: Summary: Improve configuration framework (was: Improve dynamic configuration framework) Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Attachments: design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13936: Issue Type: Umbrella (was: Bug) Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Umbrella Reporter: Apekshit Sharma Attachments: design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13936) Improve dynamic configuration framework
Apekshit Sharma created HBASE-13936: --- Summary: Improve dynamic configuration framework Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13936) Improve dynamic configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592313#comment-14592313 ] Apekshit Sharma commented on HBASE-13936: - Depending on scope of work done in this jira, link to HBASE-3909 may later be updated to duplicate/supersede. Improve dynamic configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Bug Reporter: Apekshit Sharma Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583970#comment-14583970 ] Apekshit Sharma commented on HBASE-13702: - Ping for reviews. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575315#comment-14575315 ] Apekshit Sharma commented on HBASE-13702: - Does the patch looks good? I feel like it's ready for commit. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571581#comment-14571581 ] Apekshit Sharma commented on HBASE-13702: - So, combiners and reducers in the bulk mode are executed in dry run mode too. However, TableReducer in non-bulk mode is not run in dry-mode. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568310#comment-14568310 ] Apekshit Sharma commented on HBASE-13702: - Did few runs on single node cluster. Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, single core, hyper threading = 4 Dataset 1: 1000 rows, key length = 20, #columns = 100, column length = 1k size of input csv file: 100M Dataset 2: 1 rows, key length = 20, #columns = 100, column length = 1k size of input csv file: 1G *Non Bulk Mode:* Dataset 1 Dry mode: 1 sec Non-dry mode: ~4 sec Dataset 2 Dry mode: ~10s Non dry mode: ~24 s num automatic splits: 8 Verified row count after each run. *Bulk Mode:* Dataset 2 1 rows, key length = 20, #columns = 100, column length = 1k size of input csv file: 1G dry mode: ~40 sec (table not existent on start, verified no table and output dir after run) non-dry mode: ~60 sec (table not existent on start, verified table and output dir exists after run) num automatic splits: 8 Since the runs are in order of seconds/minutes, I think we can and should test all functionality in dry-run. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563169#comment-14563169 ] Apekshit Sharma commented on HBASE-13702: - So the way I see dry-run functionality for adhoc tools like this one is check tool will run successfully on given data without making any (permanent) change to system. So ideally, users should get all errors/warning in dry run and actual run should be like butter, instead of getting stuck in a half-commit stage where some things went through and other didn't (unless it's acceptable). On practical side, I am with you if it makes sense to remove some trivial logic if it shaves of huge run-time. I don't have practical exp. of runtimes of this tool, but I would guess any processing in mapper shouldn't take much time compared to final stage of writing Put mutations to table (in non-bulk mode)/hfiles to disk(bulk mode) which dry-run already skips. If my assumptions are wrong, please let me know. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
