[jira] [Commented] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685464#comment-16685464
 ] 

Sean Busbey commented on HBASE-20586:
-

A doc jira blocked by this one sounds like a good idea.

> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce, Operability, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 1.5.0, 2.2.0
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different 
> clusters, for instance, when replication has been enabled but data existed 
> already, or due replication issues that may had caused long lags in the 
> replication.
> For secured clusters under different kerberos realms (with cross-realm 
> properly set), though, current SyncTable version would fail to authenticate 
> with the remote cluster when trying to read HashTable outputs (when 
> *sourcehashdir* is remote) and also when trying to read table data on the 
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status 
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1"; 
> destination host is: "remote-nn":8020;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1506)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
>         at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
>         at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN. 
> Once hdfs related authentication is done, it's also necessary to authenticate 
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status 
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get 
> the location
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to 
> remote-rs-host/1.1.1.2:60020
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21452) Illegal character in hbase counters group name

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685454#comment-16685454
 ] 

Sean Busbey commented on HBASE-21452:
-

Okay so release note marking it as an incompat that calls out folks looking at 
Hadoop counters (either in MapReduce jobs or Spark) will see a change in the 
names? If that sounds correct, then +1 from me for master and branch-2.

> Illegal character in hbase counters group name
> --
>
> Key: HBASE-21452
> URL: https://issues.apache.org/jira/browse/HBASE-21452
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21452.branch-2.001.patch
>
>
> Messing w/ spark counting RDD rows, spark dumps out following complaint:
> {code}
> 2018-11-07 20:03:29,132 ERROR [Executor task launch worker for task 0] 
> repl.ExecutorClassLoader: Failed to check existence of class HBase 
> Counters_en_US on REPL class server at spark://192.168.1.139:61037/classes
> java.net.URISyntaxException: Illegal character in path at index 41: 
> spark://192.168.1.139:61037/classes/HBase Counters_en_US.class
>   at java.net.URI$Parser.fail(URI.java:2848)
>   at java.net.URI$Parser.checkChars(URI.java:3021)
>   at java.net.URI$Parser.parseHierarchical(URI.java:3105)
>   at java.net.URI$Parser.parse(URI.java:3053)
>   at java.net.URI.(URI.java:588)
>   at 
> org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:328)
>   at 
> org.apache.spark.repl.ExecutorClassLoader.org$apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:95)
>   at 
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
>   at 
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
>   at 
> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:167)
>   at 
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:85)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2649)
>   at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1510)
>   at java.util.ResourceBundle.findBundle(ResourceBundle.java:1474)
>   at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1370)
>   at java.util.ResourceBundle.getBundle(ResourceBundle.java:1091)
>   at 
> org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37)
>   at 
> org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56)
>   at 
> org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterGroupName(ResourceBundles.java:77)
>   at 
> org.apache.hadoop.mapreduce.counters.CounterGroupFactory.newGroup(CounterGroupFactory.java:94)
>   at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:226)
>   at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:153)
>   at 
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl$DummyReporter.getCounter(TaskAttemptContextImpl.java:110)
>   at 
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:298)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:286)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:257)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:220)
>   at 
> org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:214)
>   at 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>   at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1837)
>   at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
>   at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
>   at 
> 

[jira] [Commented] (HBASE-21470) [hbase-connectors] Build shaded versions of the connectors libs

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685440#comment-16685440
 ] 

Sean Busbey commented on HBASE-21470:
-

the spark module should already work with the shaded hbase module(s). IIRC it 
requires the mapreduce specific one. I would expect that Kafka will end up 
being the same. If we can avoid relying on the shaded plugin again within 
hbase-connectors we should do so.

> [hbase-connectors] Build shaded versions of the connectors libs
> ---
>
> Key: HBASE-21470
> URL: https://issues.apache.org/jira/browse/HBASE-21470
> Project: HBase
>  Issue Type: Task
>  Components: build, hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Adrian Muraru
>Priority: Major
>
> For downstream users it would be helpful to generate shaded versions of the 
> connectors libs, e.g hbase-shaded-spark and hbase-shaded-kafka.
> These would ease integrating this libs in Spark/Hadoop projects where 
> transitive dependencies of the connectors libs conflict with the runtime ones



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685419#comment-16685419
 ] 

Sean Busbey commented on HBASE-20604:
-

My understanding is that a good portion of our issues in this code are caused 
by Hadoop not really defining what to expect when a client has a concurrent 
read open on a file that's still open for write. This is usually where the 
problems in our WAL reading code comes up; our replication system is relying on 
assumptions that aren't really documented anywhere.

AFAIK there's no UT because we have never been able to isolate the problem(s) 
that poke up in production around this, and trying to mock out the various 
levels of abstraction went poorly when last I tried.

> ProtobufLogReader#readNext can incorrectly loop to the same position in the 
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.9
>
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch, 
> HBASE-20604.004.patch, HBASE-20604.005.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream 
> associated to the {{FSDataInputStream}} from the WAL that we are reading. 
> Under certain conditions, e.g. when using the encryption at rest 
> ({{CryptoInputStream}}) the stream can return partial data which can cause a 
> premature EOF that cause {{inputStream.getPos()}} to return to the same 
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the 
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck 
> until the WAL is rolled and causing replication delays up to an hour in some 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685426#comment-16685426
 ] 

Sean Busbey commented on HBASE-20604:
-

Also also I would like to start the RC process for 1.2.9 this week, so it'd be 
very helpful if this critical issue didn't reopen. ;)

> ProtobufLogReader#readNext can incorrectly loop to the same position in the 
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.9
>
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch, 
> HBASE-20604.004.patch, HBASE-20604.005.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream 
> associated to the {{FSDataInputStream}} from the WAL that we are reading. 
> Under certain conditions, e.g. when using the encryption at rest 
> ({{CryptoInputStream}}) the stream can return partial data which can cause a 
> premature EOF that cause {{inputStream.getPos()}} to return to the same 
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the 
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck 
> until the WAL is rolled and causing replication delays up to an hour in some 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685425#comment-16685425
 ] 

Sean Busbey commented on HBASE-20604:
-

Also note that while we can try to isolate problems into things that Hadoop is 
doing wrong (either in CryptoInputStream or the other implementation classes), 
that project defines "what HDFS did in ~Hadoop 1" as canonical. This can 
include things that some folks might consider incorrect implementation details, 
if the project believes downstream has come to rely on it.

So to some degree we'll be stuck with defensive workarounds in our client usage 
regardless.

> ProtobufLogReader#readNext can incorrectly loop to the same position in the 
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.9
>
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch, 
> HBASE-20604.004.patch, HBASE-20604.005.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream 
> associated to the {{FSDataInputStream}} from the WAL that we are reading. 
> Under certain conditions, e.g. when using the encryption at rest 
> ({{CryptoInputStream}}) the stream can return partial data which can cause a 
> premature EOF that cause {{inputStream.getPos()}} to return to the same 
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the 
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck 
> until the WAL is rolled and causing replication delays up to an hour in some 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20952) Re-visit the WAL API

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685385#comment-16685385
 ] 

Sean Busbey commented on HBASE-20952:
-

How about if I move the job to only run weekly? Since my last comment this job 
has run 5 times, which means for about 20 hours of testing that has gotten us 
no new information.

> Re-visit the WAL API
> 
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an 
> HBase WAL API should look like. What are the primitive calls that we require 
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We 
> should also have a mind for what is happening in the Ratis LogService (but 
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and 
> backup Replication has the use-case for "tail"'ing the WAL which we 
> should provide via our new API. B doesn't do anything fancy (IIRC). We 
> should make sure all consumers are generally going to be OK with the API we 
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods 
> which were "bolted" on such as {{AbstractFSWAL}} and 
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the 
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability 
> annotations are chosen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21428) Performance issue due to userRegionLock in the ConnectionManager.

2018-11-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685285#comment-16685285
 ] 

Sean Busbey commented on HBASE-21428:
-

This should be impacting all branches, correct?

> Performance issue due to userRegionLock in the ConnectionManager.
> -
>
> Key: HBASE-21428
> URL: https://issues.apache.org/jira/browse/HBASE-21428
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: koo
>Priority: Major
>
> My service is that execute a lot of puts using HTableMultiplexer.
> After the version change, most of the requests are rejected.
> It works fine in 1.2.6.1, but there is a problem in 1.2.7.
> This issue is related with the HBASE-19260.
> Most of my threads are using a lot of time as below.
>  
> |"Worker-972" #2479 daemon prio=5 os_prio=0 tid=0x7f8cea86b000 nid=0x4c8c 
> waiting on condition [0x7f8b78104000]
>  java.lang.Thread.State: WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for <0x0005dd703b78> (a 
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>  at 
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1274)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1186)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1170)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1127)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:962)
>  at 
> org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:206)
>  at 
> org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:150)|
>  
> When I looked at the issue(HBASE-19260), I recognized the dangerous of to 
> allow accessessing multiple threads.
> However, Already create many threads with the limitations
> I think it is very inefficient to allow only one thread access.
>  
> | this.metaLookupPool = getThreadPool(
>  conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128),
>  conf.getInt("hbase.hconnection.meta.lookup.threads.core", 10),
>  "-metaLookup-shared-", new LinkedBlockingQueue());|
>  
> I want to suggest changing it that allow to have multiple locks.(but not the 
> entire thread)
> The following is pseudocode.
>  
> |int lockSize = conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128) 
> / 2;
> BlockingQueue userRegionLockQueue = new 
> LinkedBlockingQueue();
>  for (int i=0; i   userRegionLockQueue.put(new ReentrantLock());
>  }|
>  
> thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)

2018-11-08 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680442#comment-16680442
 ] 

Sean Busbey commented on HBASE-21255:
-

> I will commit it late this day if no further comments.

I don't see a +1 yet. I know you're anxious to move forward but please wait for 
a review. perhaps ping dev@hbase for volunteers

> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
> 
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
>  Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch, 
> HBASE-21225.master.002.patch, HBASE-21225.master.007.patch, 
> HBASE-21255.master.003.patch, HBASE-21255.master.004.patch, 
> HBASE-21255.master.005.patch, HBASE-21255.master.006.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
>   //TODO refactor this class
>   //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
>  * Divide origin TablePermission into three classes GlobalPermission, 
> NamespacePermission, TablePermission
>  * New UserPermission consists of a user name and a permission in one of 
> [Global, Namespace, Table]Permission.
>  * Rename TableAuthManager to AuthManager(it is IA.P), and rename some 
> methods for readability.
>  * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
>  * User cache and group cache in AuthManager is combined together.
>  * Wire proto is kept, BC should be under guarantee.
>  * Fix HBASE-21390.
>  * Resolve a small {{TODO}} global entry should be handled differently in 
> AccessControlLists



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21411) Need to document the snapshot metric data that is shown in HBase Master Web UI

2018-11-08 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21411:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

thanks for the offer [~stack]; I've got things set.

thanks again Roland!

> Need to document the snapshot metric data that is shown in HBase Master Web UI
> --
>
> Key: HBASE-21411
> URL: https://issues.apache.org/jira/browse/HBASE-21411
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Roland Teague
>Assignee: Roland Teague
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 0001-Patch-for-HBASE-21411.patch, 
> HBASE-21411.master.001.patch
>
>
> We need to add documentation into the Reference Guide for the work that was 
> done in HBASE-15415.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21454) Kill zk spew

2018-11-08 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680298#comment-16680298
 ] 

Sean Busbey commented on HBASE-21454:
-

I am enthusiastic about not seeing a bunch of ZK stuff, but server start up 
needs some source of a classpath dump. Can we add one at the same time so that 
all the services do it directly so that it won't show up in clients?

> Kill zk spew
> 
>
> Key: HBASE-21454
> URL: https://issues.apache.org/jira/browse/HBASE-21454
> Project: HBase
>  Issue Type: Bug
>  Components: logging, Zookeeper
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21454.master.001.patch
>
>
> Kill the zk spew. This is radical dropping startup listing of CLASSPATH and 
> all properties. Can dial back-in what we need after this patch goes in.
> I get spew each time I run a little command in spark-shell. Annoying. Always 
> been annoying in all logs.
> More might be needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21452) Illegal character in hbase counters group name

2018-11-08 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680356#comment-16680356
 ] 

Sean Busbey commented on HBASE-21452:
-

this will impact any existing MR users, right?

> Illegal character in hbase counters group name
> --
>
> Key: HBASE-21452
> URL: https://issues.apache.org/jira/browse/HBASE-21452
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21452.branch-2.001.patch
>
>
> Messing w/ spark counting RDD rows, spark dumps out following complaint:
> {code}
> 2018-11-07 20:03:29,132 ERROR [Executor task launch worker for task 0] 
> repl.ExecutorClassLoader: Failed to check existence of class HBase 
> Counters_en_US on REPL class server at spark://192.168.1.139:61037/classes
> java.net.URISyntaxException: Illegal character in path at index 41: 
> spark://192.168.1.139:61037/classes/HBase Counters_en_US.class
>   at java.net.URI$Parser.fail(URI.java:2848)
>   at java.net.URI$Parser.checkChars(URI.java:3021)
>   at java.net.URI$Parser.parseHierarchical(URI.java:3105)
>   at java.net.URI$Parser.parse(URI.java:3053)
>   at java.net.URI.(URI.java:588)
>   at 
> org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:328)
>   at 
> org.apache.spark.repl.ExecutorClassLoader.org$apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:95)
>   at 
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
>   at 
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
>   at 
> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:167)
>   at 
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:85)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2649)
>   at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1510)
>   at java.util.ResourceBundle.findBundle(ResourceBundle.java:1474)
>   at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1370)
>   at java.util.ResourceBundle.getBundle(ResourceBundle.java:1091)
>   at 
> org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37)
>   at 
> org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56)
>   at 
> org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterGroupName(ResourceBundles.java:77)
>   at 
> org.apache.hadoop.mapreduce.counters.CounterGroupFactory.newGroup(CounterGroupFactory.java:94)
>   at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:226)
>   at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:153)
>   at 
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl$DummyReporter.getCounter(TaskAttemptContextImpl.java:110)
>   at 
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:298)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:286)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:257)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133)
>   at 
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:220)
>   at 
> org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:214)
>   at 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
>   at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1837)
>   at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
>   at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
>   at 
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
>   at 
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
>   at 

[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split

2018-11-08 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680322#comment-16680322
 ] 

Sean Busbey commented on HBASE-21355:
-

awesome, thanks for the follow up.

> HStore's storeSize is calculated repeatedly which causing the confusing 
> region split 
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch, 
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster,  we 
> found that the region will be inexplicably split.  
> We use the default ConstantSizeRegionSplitPolicy and 
> hbase.hregion.max.filesize=40G,but  the region will be split even if its 
> bytes size is less than 40G(only ~6G). 
> Checked the code, I found that the following path  will  accumulate the 
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
>   -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh 
> the store files.
> Some comment here,  I move the  storeSize calculation out of loadStoreFiles() 
> method, because the secondary read replica's refreshStoreFiles() will also 
> use loadStoreFiles() to refresh its store files and update the storeSize in 
> the completeCompaction(..) in the final (just like compaction.) , so no need 
> calculate the storeSize twice.. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21458) Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty

2018-11-08 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680301#comment-16680301
 ] 

Sean Busbey commented on HBASE-21458:
-

can you post up a log of the message you're seeing? run in debug mode maybe? do 
these extra jars end up in the classpath we hand back to the caller?

> Error: Could not find or load main class 
> org.apache.hadoop.hbase.util.GetJavaProperty
> -
>
> Key: HBASE-21458
> URL: https://issues.apache.org/jira/browse/HBASE-21458
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-21458.branch-2.1.001.patch
>
>
> I get this when I run bin/hbase classpath whether a built checkout or an 
> undone tarball.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21355) HStore's storeSize is calculated repeatedly which causing the confusing region split

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678681#comment-16678681
 ] 

Sean Busbey commented on HBASE-21355:
-

did this not impact branch-1.2 or was it just overlooked?

> HStore's storeSize is calculated repeatedly which causing the confusing 
> region split 
> -
>
> Key: HBASE-21355
> URL: https://issues.apache.org/jira/browse/HBASE-21355
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21355.addendum.patch, HBASE-21355.addendum.patch, 
> HBASE-21355.branch-1.patch, HBASE-21355.v1.patch
>
>
> When testing the branch-2's write performance in our internal cluster,  we 
> found that the region will be inexplicably split.  
> We use the default ConstantSizeRegionSplitPolicy and 
> hbase.hregion.max.filesize=40G,but  the region will be split even if its 
> bytes size is less than 40G(only ~6G). 
> Checked the code, I found that the following path  will  accumulate the 
> store's storeSize to a very big value, because the path has no reset..
> {code}
> RsRpcServices#getRegionInfo
>   -> HRegion#isMergeable
>-> HRegion#hasReferences
> -> HStore#hasReferences
> -> HStore#openStoreFiles
> {code}
> BTW, we seems forget to maintain the read replica's storeSize when refresh 
> the store files.
> Some comment here,  I move the  storeSize calculation out of loadStoreFiles() 
> method, because the secondary read replica's refreshStoreFiles() will also 
> use loadStoreFiles() to refresh its store files and update the storeSize in 
> the completeCompaction(..) in the final (just like compaction.) , so no need 
> calculate the storeSize twice.. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21411) Need to document the snapshot metric data that is shown in HBase Master Web UI

2018-11-07 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21411:

Status: Patch Available  (was: Open)

> Need to document the snapshot metric data that is shown in HBase Master Web UI
> --
>
> Key: HBASE-21411
> URL: https://issues.apache.org/jira/browse/HBASE-21411
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Roland Teague
>Assignee: Roland Teague
>Priority: Major
> Attachments: HBASE-21411.master.001.patch
>
>
> We need to add documentation into the Reference Guide for the work that was 
> done in HBASE-15415.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21411) Need to document the snapshot metric data that is shown in HBase Master Web UI

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678596#comment-16678596
 ] 

Sean Busbey commented on HBASE-21411:
-

please use {{git format-patch}} to create your patch so that it will include 
authorship information as you'd like to have it appear.

> Need to document the snapshot metric data that is shown in HBase Master Web UI
> --
>
> Key: HBASE-21411
> URL: https://issues.apache.org/jira/browse/HBASE-21411
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Roland Teague
>Assignee: Roland Teague
>Priority: Major
> Attachments: HBASE-21411.master.001.patch
>
>
> We need to add documentation into the Reference Guide for the work that was 
> done in HBASE-15415.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HBASE-21411) Need to document the snapshot metric data that is shown in HBase Master Web UI

2018-11-07 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reassigned HBASE-21411:
---

Assignee: Roland Teague

Thanks for the patch Roland! I've added you to the contributor role in JIRA so 
you ought to be able to assign issues to yourself now (as well as mark them 
"patch available" for qabot checking and review)

> Need to document the snapshot metric data that is shown in HBase Master Web UI
> --
>
> Key: HBASE-21411
> URL: https://issues.apache.org/jira/browse/HBASE-21411
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Roland Teague
>Assignee: Roland Teague
>Priority: Major
> Attachments: HBASE-21411.master.001.patch
>
>
> We need to add documentation into the Reference Guide for the work that was 
> done in HBASE-15415.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20952) Re-visit the WAL API

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678495#comment-16678495
 ] 

Sean Busbey commented on HBASE-20952:
-

Can we wait to make the branch until there are commits for it? Or wait to run 
the tests until then?

> Re-visit the WAL API
> 
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an 
> HBase WAL API should look like. What are the primitive calls that we require 
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We 
> should also have a mind for what is happening in the Ratis LogService (but 
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and 
> backup Replication has the use-case for "tail"'ing the WAL which we 
> should provide via our new API. B doesn't do anything fancy (IIRC). We 
> should make sure all consumers are generally going to be OK with the API we 
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods 
> which were "bolted" on such as {{AbstractFSWAL}} and 
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the 
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability 
> annotations are chosen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678398#comment-16678398
 ] 

Sean Busbey commented on HBASE-20604:
-

+1 on v5 pending qabot

> ProtobufLogReader#readNext can incorrectly loop to the same position in the 
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch, 
> HBASE-20604.004.patch, HBASE-20604.005.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream 
> associated to the {{FSDataInputStream}} from the WAL that we are reading. 
> Under certain conditions, e.g. when using the encryption at rest 
> ({{CryptoInputStream}}) the stream can return partial data which can cause a 
> premature EOF that cause {{inputStream.getPos()}} to return to the same 
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the 
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck 
> until the WAL is rolled and causing replication delays up to an hour in some 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21443) [hbase-connectors] Purge hbase-* modules from core now they've been moved to hbase-connectors

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678310#comment-16678310
 ] 

Sean Busbey commented on HBASE-21443:
-

so I'm +1 on either the patch w/o the scalatools plugin or the one that 
includes it (though in the case of the latter I'll probably file a jira to 
remove it afterwards)

> [hbase-connectors] Purge hbase-* modules from core now they've been moved to 
> hbase-connectors
> -
>
> Key: HBASE-21443
> URL: https://issues.apache.org/jira/browse/HBASE-21443
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-connectors, spark
>Affects Versions: 3.0.0, 2.2.0
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21443.master.001.patch, 
> HBASE-21443.master.002.patch, HBASE-21443.master.002.patch
>
>
> The parent copied the spark modules over to hbase-connectors. Here we purge 
> them from hbase core repo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-15557) Add guidance on HashTable/SyncTable to the RefGuide

2018-11-07 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-15557:

Summary: Add guidance on HashTable/SyncTable to the RefGuide  (was: 
document SyncTable in ref guide)

> Add guidance on HashTable/SyncTable to the RefGuide
> ---
>
> Key: HBASE-15557
> URL: https://issues.apache.org/jira/browse/HBASE-15557
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.2.0
>Reporter: Sean Busbey
>Assignee: Wellington Chevreuil
>Priority: Critical
> Attachments: HBASE-15557.master.001.patch, 
> HBASE-15557.master.002.patch
>
>
> The docs for SyncTable are insufficient. Brief description from [~davelatham] 
> HBASE-13639 comment:
> {quote}
> Sorry for the lack of better documentation, Abhishek Soni. Thanks for 
> bringing it up. I'll try to provide a better explanation. You may have 
> already seen it, but if not, the design doc linked in the description above 
> may also give you some better clues as to how it should be used.
> Briefly, the feature is intended to start with a pair of tables in remote 
> clusters that are already substantially similar and make them identical by 
> comparing hashes of the data and copying only the diffs instead of having to 
> copy the entire table. So it is targeted at a very specific use case (with 
> some work it could generalize to cover things like CopyTable and 
> VerifyRepliaction but it's not there yet). To use it, you choose one table to 
> be the "source", and the other table is the "target". After the process is 
> complete the target table should end up being identical to the source table.
> In the source table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.HashTable and pass it the name of the 
> source table and an output directory in HDFS. HashTable will scan the source 
> table, break the data up into row key ranges (default of 8kB per range) and 
> produce a hash of the data for each range.
> Make the hashes available to the target cluster - I'd recommend using DistCp 
> to copy it across.
> In the target table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.SyncTable and pass it the directory where 
> you put the hashes, and the names of the source and destination tables. You 
> will likely also need to specify the source table's ZK quorum via the 
> --sourcezkcluster option. SyncTable will then read the hash information, and 
> compute the hashes of the same row ranges for the target table. For any row 
> range where the hash fails to match, it will open a remote scanner to the 
> source table, read the data for that range, and do Puts and Deletes to the 
> target table to update it to match the source.
> I hope that clarifies it a bit. Let me know if you need a hand. If anyone 
> wants to work on getting some documentation into the book, I can try to write 
> some more but would love a hand on turning it into an actual book patch.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678295#comment-16678295
 ] 

Sean Busbey commented on HBASE-20586:
-

I agree that we don't have the needed infra to have a test for this right now. 
I would like whoever commits it to try running the change as well, especially 
given that it's been ~6 months since it was submitted. I'll try to make time 
next week.

> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce, Operability, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 1.5.0, 2.2.0
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different 
> clusters, for instance, when replication has been enabled but data existed 
> already, or due replication issues that may had caused long lags in the 
> replication.
> For secured clusters under different kerberos realms (with cross-realm 
> properly set), though, current SyncTable version would fail to authenticate 
> with the remote cluster when trying to read HashTable outputs (when 
> *sourcehashdir* is remote) and also when trying to read table data on the 
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status 
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1"; 
> destination host is: "remote-nn":8020;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1506)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
>         at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
>         at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN. 
> Once hdfs related authentication is done, it's also necessary to authenticate 
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status 
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get 
> the location
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to 
> remote-rs-host/1.1.1.2:60020
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-15557) Add guidance on HashTable/SyncTable to the RefGuide

2018-11-07 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-15557:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

merged. Thanks again [~wchevreuil] this is a great doc addition!

maybe for follow-on, this bit sounds like an error condition we should detect?

{code}
+.Set sourcezkcluster to the actual source cluster ZK quorum
+[NOTE]
+
+Although not required, if sourcezkcluster is not set, SyncTable will connect 
to local HBase cluster for both source and target,
+which does not give any meaningful result.
{code}


> Add guidance on HashTable/SyncTable to the RefGuide
> ---
>
> Key: HBASE-15557
> URL: https://issues.apache.org/jira/browse/HBASE-15557
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.2.0
>Reporter: Sean Busbey
>Assignee: Wellington Chevreuil
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-15557.master.001.patch, 
> HBASE-15557.master.002.patch
>
>
> The docs for SyncTable are insufficient. Brief description from [~davelatham] 
> HBASE-13639 comment:
> {quote}
> Sorry for the lack of better documentation, Abhishek Soni. Thanks for 
> bringing it up. I'll try to provide a better explanation. You may have 
> already seen it, but if not, the design doc linked in the description above 
> may also give you some better clues as to how it should be used.
> Briefly, the feature is intended to start with a pair of tables in remote 
> clusters that are already substantially similar and make them identical by 
> comparing hashes of the data and copying only the diffs instead of having to 
> copy the entire table. So it is targeted at a very specific use case (with 
> some work it could generalize to cover things like CopyTable and 
> VerifyRepliaction but it's not there yet). To use it, you choose one table to 
> be the "source", and the other table is the "target". After the process is 
> complete the target table should end up being identical to the source table.
> In the source table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.HashTable and pass it the name of the 
> source table and an output directory in HDFS. HashTable will scan the source 
> table, break the data up into row key ranges (default of 8kB per range) and 
> produce a hash of the data for each range.
> Make the hashes available to the target cluster - I'd recommend using DistCp 
> to copy it across.
> In the target table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.SyncTable and pass it the directory where 
> you put the hashes, and the names of the source and destination tables. You 
> will likely also need to specify the source table's ZK quorum via the 
> --sourcezkcluster option. SyncTable will then read the hash information, and 
> compute the hashes of the same row ranges for the target table. For any row 
> range where the hash fails to match, it will open a remote scanner to the 
> source table, read the data for that range, and do Puts and Deletes to the 
> target table to update it to match the source.
> I hope that clarifies it a bit. Let me know if you need a hand. If anyone 
> wants to work on getting some documentation into the book, I can try to write 
> some more but would love a hand on turning it into an actual book patch.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21443) [hbase-connectors] Purge hbase-* modules from core now they've been moved to hbase-connectors

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678269#comment-16678269
 ] 

Sean Busbey commented on HBASE-21443:
-

Peter has it exactly correct. It's just that Yetus sees scala files have 
changed and the scaladoc test is around (because we tell it to use "all" tests 
basically) so it tries to run the test on the presumption that a maven project 
with scala files will have configured the scala plugins.

Yetus would be more robust here if it referred to the fully qualified maven 
plugin name. we should file a bug for that.

In general I think the workaround is to treat this as a false error and ignore 
it.

> [hbase-connectors] Purge hbase-* modules from core now they've been moved to 
> hbase-connectors
> -
>
> Key: HBASE-21443
> URL: https://issues.apache.org/jira/browse/HBASE-21443
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-connectors, spark
>Affects Versions: 3.0.0, 2.2.0
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21443.master.001.patch, 
> HBASE-21443.master.002.patch, HBASE-21443.master.002.patch
>
>
> The parent copied the spark modules over to hbase-connectors. Here we purge 
> them from hbase core repo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20952) Re-visit the WAL API

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678265#comment-16678265
 ] 

Sean Busbey commented on HBASE-20952:
-

{quote}
>From https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/41// , we 
>can see that TestIncrementalBackupWithBulkLoad failed for hadoop3 build.
This is known issue - see HADOOP-15850.

Other than that, the build in HBASE-20952 branch is quite normal.
{quote}

Please do what it takes to get passing builds. If there is a known cause of 
failure, can a JUnit Assume be used to disable the test on known-bad versions? 
does the branch require a particular version of Hadoop? If so, why doesn't it 
have that expressed in the pom?

> Re-visit the WAL API
> 
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an 
> HBase WAL API should look like. What are the primitive calls that we require 
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We 
> should also have a mind for what is happening in the Ratis LogService (but 
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and 
> backup Replication has the use-case for "tail"'ing the WAL which we 
> should provide via our new API. B doesn't do anything fancy (IIRC). We 
> should make sure all consumers are generally going to be OK with the API we 
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods 
> which were "bolted" on such as {{AbstractFSWAL}} and 
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the 
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability 
> annotations are chosen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

2018-11-07 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678237#comment-16678237
 ] 

Sean Busbey commented on HBASE-20604:
-

log message on line 422 still needs to include a mention of a malformed edit. 
checkstyle is close but still off by a bit.

> ProtobufLogReader#readNext can incorrectly loop to the same position in the 
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch, 
> HBASE-20604.004.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream 
> associated to the {{FSDataInputStream}} from the WAL that we are reading. 
> Under certain conditions, e.g. when using the encryption at rest 
> ({{CryptoInputStream}}) the stream can return partial data which can cause a 
> premature EOF that cause {{inputStream.getPos()}} to return to the same 
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the 
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck 
> until the WAL is rolled and causing replication delays up to an hour in some 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677318#comment-16677318
 ] 

Sean Busbey commented on HBASE-20604:
-

{code}
@@ -416,7 +420,15 @@ public class ProtobufLogReader extends ReaderBase {
 if (LOG.isTraceEnabled()) {
   LOG.trace("Encountered a malformed edit, seeking back to last good 
position in file, from "+ inputStream.getPos()+" to " + originalPosition, eof);
 }
-seekOnFs(originalPosition);
+// If stuck at the same place and we got and exception, lets go back 
at the beginning.
+if (inputStream.getPos() == originalPosition && resetPosition) {
+  if (LOG.isTraceEnabled()) {
+LOG.trace("Seeking to the beginning of the WAL, current position " 
+ originalPosition + " is the same as the original position.");
+  }
+  seekOnFs(0);
+} else {
+  seekOnFs(originalPosition);
+}
{code}

The {{LOG.trace}} block just before this addition should be inside of the 
{{else}} clause that's added, because currently in the "reset to start" case 
we're effectively duplicating the TRACE messages.

After the above, the {{LOG.trace}} message provided when we seek to the start 
should include in the why ("original and current positions match") that we got 
a malformed edit.

With those two changes and the long line from checkstyle corrected, I'm +1.

> ProtobufLogReader#readNext can incorrectly loop to the same position in the 
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Attachments: HBASE-20604.002.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream 
> associated to the {{FSDataInputStream}} from the WAL that we are reading. 
> Under certain conditions, e.g. when using the encryption at rest 
> ({{CryptoInputStream}}) the stream can return partial data which can cause a 
> premature EOF that cause {{inputStream.getPos()}} to return to the same 
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the 
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck 
> until the WAL is rolled and causing replication delays up to an hour in some 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (HBASE-21396) Create 2.1.1 release

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-21396.
-
Resolution: Fixed

all set now, re-resolving.

> Create 2.1.1 release
> 
>
> Key: HBASE-21396
> URL: https://issues.apache.org/jira/browse/HBASE-21396
> Project: HBase
>  Issue Type: Task
>  Components: rm
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (HBASE-21442) Update branch-2.1 for next development cycle

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-21442.
-
Resolution: Fixed

> Update branch-2.1 for next development cycle
> 
>
> Key: HBASE-21442
> URL: https://issues.apache.org/jira/browse/HBASE-21442
> Project: HBase
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 2.1.1
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.1.2
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-21442) Update branch-2.1 for next development cycle

2018-11-06 Thread Sean Busbey (JIRA)
Sean Busbey created HBASE-21442:
---

 Summary: Update branch-2.1 for next development cycle
 Key: HBASE-21442
 URL: https://issues.apache.org/jira/browse/HBASE-21442
 Project: HBase
  Issue Type: Sub-task
  Components: build
Affects Versions: 2.1.1
Reporter: Sean Busbey
Assignee: Sean Busbey
 Fix For: 2.1.2






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Work started] (HBASE-21442) Update branch-2.1 for next development cycle

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-21442 started by Sean Busbey.
---
> Update branch-2.1 for next development cycle
> 
>
> Key: HBASE-21442
> URL: https://issues.apache.org/jira/browse/HBASE-21442
> Project: HBase
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 2.1.1
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.1.2
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Reopened] (HBASE-21396) Create 2.1.1 release

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reopened HBASE-21396:
-

> Create 2.1.1 release
> 
>
> Key: HBASE-21396
> URL: https://issues.apache.org/jira/browse/HBASE-21396
> Project: HBase
>  Issue Type: Task
>  Components: rm
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21396) Create 2.1.1 release

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677144#comment-16677144
 ] 

Sean Busbey commented on HBASE-21396:
-

branch-2.1 still has a version of 2.1.1 instead of 2.1.2-SNAPSHOT. want the fix 
in a subtask of here, a new top level issue, or just referencing this jira?

> Create 2.1.1 release
> 
>
> Key: HBASE-21396
> URL: https://issues.apache.org/jira/browse/HBASE-21396
> Project: HBase
>  Issue Type: Task
>  Components: rm
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-15557) document SyncTable in ref guide

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677001#comment-16677001
 ] 

Sean Busbey commented on HBASE-15557:
-

looks great. I'm all set to push this. [~wchevreuil] is the email address on 
the current patch the one you want attribution to go to?

> document SyncTable in ref guide
> ---
>
> Key: HBASE-15557
> URL: https://issues.apache.org/jira/browse/HBASE-15557
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.2.0
>Reporter: Sean Busbey
>Assignee: Wellington Chevreuil
>Priority: Critical
> Attachments: HBASE-15557.master.001.patch
>
>
> The docs for SyncTable are insufficient. Brief description from [~davelatham] 
> HBASE-13639 comment:
> {quote}
> Sorry for the lack of better documentation, Abhishek Soni. Thanks for 
> bringing it up. I'll try to provide a better explanation. You may have 
> already seen it, but if not, the design doc linked in the description above 
> may also give you some better clues as to how it should be used.
> Briefly, the feature is intended to start with a pair of tables in remote 
> clusters that are already substantially similar and make them identical by 
> comparing hashes of the data and copying only the diffs instead of having to 
> copy the entire table. So it is targeted at a very specific use case (with 
> some work it could generalize to cover things like CopyTable and 
> VerifyRepliaction but it's not there yet). To use it, you choose one table to 
> be the "source", and the other table is the "target". After the process is 
> complete the target table should end up being identical to the source table.
> In the source table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.HashTable and pass it the name of the 
> source table and an output directory in HDFS. HashTable will scan the source 
> table, break the data up into row key ranges (default of 8kB per range) and 
> produce a hash of the data for each range.
> Make the hashes available to the target cluster - I'd recommend using DistCp 
> to copy it across.
> In the target table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.SyncTable and pass it the directory where 
> you put the hashes, and the names of the source and destination tables. You 
> will likely also need to specify the source table's ZK quorum via the 
> --sourcezkcluster option. SyncTable will then read the hash information, and 
> compute the hashes of the same row ranges for the target table. For any row 
> range where the hash fails to match, it will open a remote scanner to the 
> source table, read the data for that range, and do Puts and Deletes to the 
> target table to update it to match the source.
> I hope that clarifies it a bit. Let me know if you need a hand. If anyone 
> wants to work on getting some documentation into the book, I can try to write 
> some more but would love a hand on turning it into an actual book patch.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20586:

Priority: Major  (was: Minor)

> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce, Operability, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 1.5.0, 2.2.0
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different 
> clusters, for instance, when replication has been enabled but data existed 
> already, or due replication issues that may had caused long lags in the 
> replication.
> For secured clusters under different kerberos realms (with cross-realm 
> properly set), though, current SyncTable version would fail to authenticate 
> with the remote cluster when trying to read HashTable outputs (when 
> *sourcehashdir* is remote) and also when trying to read table data on the 
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status 
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1"; 
> destination host is: "remote-nn":8020;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1506)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
>         at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
>         at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN. 
> Once hdfs related authentication is done, it's also necessary to authenticate 
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status 
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get 
> the location
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to 
> remote-rs-host/1.1.1.2:60020
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20586:

Component/s: Operability

> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce, Operability, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 1.5.0, 2.2.0
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different 
> clusters, for instance, when replication has been enabled but data existed 
> already, or due replication issues that may had caused long lags in the 
> replication.
> For secured clusters under different kerberos realms (with cross-realm 
> properly set), though, current SyncTable version would fail to authenticate 
> with the remote cluster when trying to read HashTable outputs (when 
> *sourcehashdir* is remote) and also when trying to read table data on the 
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status 
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1"; 
> destination host is: "remote-nn":8020;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1506)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
>         at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
>         at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN. 
> Once hdfs related authentication is done, it's also necessary to authenticate 
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status 
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get 
> the location
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to 
> remote-rs-host/1.1.1.2:60020
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676992#comment-16676992
 ] 

Sean Busbey commented on HBASE-20586:
-

What are we waiting on here? basically for a committer to set up two clusters 
with different Kerberos Realms and try it out?

> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 1.5.0, 2.2.0
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different 
> clusters, for instance, when replication has been enabled but data existed 
> already, or due replication issues that may had caused long lags in the 
> replication.
> For secured clusters under different kerberos realms (with cross-realm 
> properly set), though, current SyncTable version would fail to authenticate 
> with the remote cluster when trying to read HashTable outputs (when 
> *sourcehashdir* is remote) and also when trying to read table data on the 
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status 
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1"; 
> destination host is: "remote-nn":8020;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1506)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
>         at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
>         at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN. 
> Once hdfs related authentication is done, it's also necessary to authenticate 
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status 
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get 
> the location
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to 
> remote-rs-host/1.1.1.2:60020
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20586:

Fix Version/s: 2.2.0
   1.5.0

> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 1.5.0, 2.2.0
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different 
> clusters, for instance, when replication has been enabled but data existed 
> already, or due replication issues that may had caused long lags in the 
> replication.
> For secured clusters under different kerberos realms (with cross-realm 
> properly set), though, current SyncTable version would fail to authenticate 
> with the remote cluster when trying to read HashTable outputs (when 
> *sourcehashdir* is remote) and also when trying to read table data on the 
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status 
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1"; 
> destination host is: "remote-nn":8020;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1506)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
>         at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
>         at 
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
>         at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN. 
> Once hdfs related authentication is done, it's also necessary to authenticate 
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status 
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get 
> the location
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at 
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to 
> remote-rs-host/1.1.1.2:60020
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase 
> cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-15557) document SyncTable in ref guide

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-15557:

Status: Patch Available  (was: Open)

moved to patch available so I can check out the rendered version. Thanks 
[~wchevreuil]!

> document SyncTable in ref guide
> ---
>
> Key: HBASE-15557
> URL: https://issues.apache.org/jira/browse/HBASE-15557
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.2.0
>Reporter: Sean Busbey
>Assignee: Wellington Chevreuil
>Priority: Critical
> Attachments: HBASE-15557.master.001.patch
>
>
> The docs for SyncTable are insufficient. Brief description from [~davelatham] 
> HBASE-13639 comment:
> {quote}
> Sorry for the lack of better documentation, Abhishek Soni. Thanks for 
> bringing it up. I'll try to provide a better explanation. You may have 
> already seen it, but if not, the design doc linked in the description above 
> may also give you some better clues as to how it should be used.
> Briefly, the feature is intended to start with a pair of tables in remote 
> clusters that are already substantially similar and make them identical by 
> comparing hashes of the data and copying only the diffs instead of having to 
> copy the entire table. So it is targeted at a very specific use case (with 
> some work it could generalize to cover things like CopyTable and 
> VerifyRepliaction but it's not there yet). To use it, you choose one table to 
> be the "source", and the other table is the "target". After the process is 
> complete the target table should end up being identical to the source table.
> In the source table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.HashTable and pass it the name of the 
> source table and an output directory in HDFS. HashTable will scan the source 
> table, break the data up into row key ranges (default of 8kB per range) and 
> produce a hash of the data for each range.
> Make the hashes available to the target cluster - I'd recommend using DistCp 
> to copy it across.
> In the target table's cluster, run 
> org.apache.hadoop.hbase.mapreduce.SyncTable and pass it the directory where 
> you put the hashes, and the names of the source and destination tables. You 
> will likely also need to specify the source table's ZK quorum via the 
> --sourcezkcluster option. SyncTable will then read the hash information, and 
> compute the hashes of the same row ranges for the target table. For any row 
> range where the hash fails to match, it will open a remote scanner to the 
> source table, read the data for that range, and do Puts and Deletes to the 
> target table to update it to match the source.
> I hope that clarifies it a bit. Let me know if you need a hand. If anyone 
> wants to work on getting some documentation into the book, I can try to write 
> some more but would love a hand on turning it into an actual book patch.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21432) [hbase-connectors] Add Apache Yetus integration for hbase-connectors repository

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676872#comment-16676872
 ] 

Sean Busbey commented on HBASE-21432:
-

the new precommit job can just take PR urls, because Yetus already knows how to 
handle them.

this is the github API for getting PRs:

https://developer.github.com/v3/pulls/

e.g. this is the set of open PRs sorted by most-recently-updated-first:

https://api.github.com/repos/apache/hbase-connectors/pulls?sort=updated=DESC

> [hbase-connectors] Add Apache Yetus integration for hbase-connectors 
> repository 
> 
>
> Key: HBASE-21432
> URL: https://issues.apache.org/jira/browse/HBASE-21432
> Project: HBase
>  Issue Type: Task
>  Components: build, hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Peter Somogyi
>Priority: Major
>
> Add automated testing for pull requests and patch files created for 
> hbase-connectors repository. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21432) [hbase-connectors] Add Apache Yetus integration for hbase-connectors repository

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676869#comment-16676869
 ] 

Sean Busbey commented on HBASE-21432:
-

Nah. Travis is overloaded for ASF projects already and I don't have a working 
example for using Yetus with it.

We should add a new ASF Jenkins job that goes over a list of github repos and 
does the equivalent tracking of PRs to test that the yetus jira polling things 
does:

https://github.com/apache/yetus/blob/master/precommit/jenkins/jenkins-admin.py

(or maybe update the yetus script if that's easier)

afterwards we'll have

* PreCommit-HBASE-Build - the current tester for our main repo
* PreCommit-HBASE-CONNECTORS-Build - the tester for the hbase-connectors repo

and we can expand as needed for the other repos.

> [hbase-connectors] Add Apache Yetus integration for hbase-connectors 
> repository 
> 
>
> Key: HBASE-21432
> URL: https://issues.apache.org/jira/browse/HBASE-21432
> Project: HBase
>  Issue Type: Task
>  Components: build, hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Peter Somogyi
>Priority: Major
>
> Add automated testing for pull requests and patch files created for 
> hbase-connectors repository. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21328) add HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP switch to hbase-env.sh

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21328:

Summary: add HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP switch to hbase-env.sh  
(was: why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?)

> add HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP switch to hbase-env.sh
> 
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation, Operability
>Reporter: Nick.han
>Assignee: Nick.han
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21328.master.001.patch, 
> HBASE-21328.master.002.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21328:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

pushed to relevant branches. Thanks for the docs contribution!

> why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
> --
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation, Operability
>Reporter: Nick.han
>Assignee: Nick.han
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21328.master.001.patch, 
> HBASE-21328.master.002.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21247:

Affects Version/s: 2.2.0
   3.0.0
   2.1.1
   2.0.2

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.2
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v11.txt, 
> 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 
> 21247.v6.txt, 21247.v7.txt, 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676837#comment-16676837
 ] 

Sean Busbey commented on HBASE-21247:
-

Also Ted please update the subject to match the problem that was fixed.

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.2
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v11.txt, 
> 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 
> 21247.v6.txt, 21247.v7.txt, 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-11-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21247:

Component/s: wal

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v11.txt, 
> 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 
> 21247.v6.txt, 21247.v7.txt, 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-11-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676833#comment-16676833
 ] 

Sean Busbey commented on HBASE-21247:
-

Yes, it should go to all branches that got HBASE-20856 

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v11.txt, 
> 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 
> 21247.v6.txt, 21247.v7.txt, 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21432) [hbase-connectors] Add Apache Yetus integration for hbase-connectors repository

2018-11-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675886#comment-16675886
 ] 

Sean Busbey commented on HBASE-21432:
-

How about just PRs? Right now yetus has no notion of multiple repositories 
corresponding to a single JIRA tracker, so if we want patches on JIRA to work 
we'll need to add our own logic for it.

By comparison, a job to periodically check for open PRs and then test them 
should be much simpler.

> [hbase-connectors] Add Apache Yetus integration for hbase-connectors 
> repository 
> 
>
> Key: HBASE-21432
> URL: https://issues.apache.org/jira/browse/HBASE-21432
> Project: HBase
>  Issue Type: Task
>  Components: build, hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Peter Somogyi
>Priority: Major
>
> Add automated testing for pull requests and patch files created for 
> hbase-connectors repository. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)

2018-11-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675484#comment-16675484
 ] 

Sean Busbey commented on HBASE-21255:
-

the precommit failures were probably the openjdk / surefire conflict that got 
handled in HBASE-21417 (and it looks like they're clear now).

please fix checkstyle.

> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
> 
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
>  Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch, 
> HBASE-21225.master.002.patch, HBASE-21255.master.003.patch, 
> HBASE-21255.master.004.patch, HBASE-21255.master.005.patch, 
> HBASE-21255.master.006.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
>   //TODO refactor this class
>   //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
>  * Divide origin TablePermission into three classes GlobalPermission, 
> NamespacePermission, TablePermission
>  * New UserPermission consists of a user name and a permission in one of 
> [Global, Namespace, Table]Permission.
>  * Rename TableAuthManager to AuthManager(it is IA.P), and rename some 
> methods for readability.
>  * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
>  * User cache and group cache in AuthManager is combined together.
>  * Wire proto is kept, BC should be under guarantee.
>  * Fix HBASE-21390.
>  * Resolve a small {{TODO}} global entry should be handled differently in 
> AccessControlLists



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-11-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675358#comment-16675358
 ] 

Sean Busbey commented on HBASE-21247:
-

+1

nit:
{code}
129   public Class getProviderClass(String key,
130   Class defaultVal) {
131 if (conf.get(key) == null) {
132   return conf.getClass(key, defaultVal, WALProvider.class);
133 }
{code}

I think this extra call to {{getClass}} instead of just returning 
{{defaultVal}} (since we already know {{key}} doesn't map to a value) is 
confusing, but I suppose this is _technically_ more correct in case we have 
some kind of special {{Configuration}} implementation that has its own opinion 
about how fallback to the passed default class works.

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v11.txt, 
> 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 
> 21247.v6.txt, 21247.v7.txt, 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20952) Re-visit the WAL API

2018-11-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675317#comment-16675317
 ] 

Sean Busbey commented on HBASE-20952:
-

Don't see much movement on the feature branch for this. I'd like to disable the 
nightly test job for it until development picks up again.

Please shout if you'd prefer it stay on. If so, please plan to clean up 
failures.

> Re-visit the WAL API
> 
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an 
> HBase WAL API should look like. What are the primitive calls that we require 
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We 
> should also have a mind for what is happening in the Ratis LogService (but 
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and 
> backup Replication has the use-case for "tail"'ing the WAL which we 
> should provide via our new API. B doesn't do anything fancy (IIRC). We 
> should make sure all consumers are generally going to be OK with the API we 
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods 
> which were "bolted" on such as {{AbstractFSWAL}} and 
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the 
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability 
> annotations are chosen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21430) [hbase-connectors] Move hbase-spark* modules to hbase-connectors repo

2018-11-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675265#comment-16675265
 ] 

Sean Busbey commented on HBASE-21430:
-

just send them to the existing {{issues@}} list.

I do think its own repo would give us more options around how we maintain 
compatibility with various spark release lines. I'm up for whatever folks want 
to do if they're doing the work though. :)

> [hbase-connectors] Move hbase-spark* modules to hbase-connectors repo
> -
>
> Key: HBASE-21430
> URL: https://issues.apache.org/jira/browse/HBASE-21430
> Project: HBase
>  Issue Type: Bug
>  Components: hbase-connectors, spark
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> Exploring moving the spark modules out of core hbase and into 
> hbase-connectors. Perhaps spark is deserving of its own repo (I think 
> [~busbey] was on about this) but meantime, experimenting w/ having it out in 
> hbase-connectors.
> Here is thread on spark integration 
> https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21427) Make it so flakie's and nightlies build rates are configurable from jenkins Config page

2018-11-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675254#comment-16675254
 ] 

Sean Busbey commented on HBASE-21427:
-

related / not related I agree that nightly should be run once / day. the flaky 
job though loses a lot of utility if it isn't running very often. By their 
definition these are tests that we need multiple runs to see failures and 
multiple parallel runs for catch-up are a feature rather than a bug.

> Make it so flakie's and nightlies build rates are configurable from jenkins 
> Config page
> ---
>
> Key: HBASE-21427
> URL: https://issues.apache.org/jira/browse/HBASE-21427
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: stack
>Priority: Major
>
> Request is that rather than have to change code whenever we want to change 
> build rates, would be easier doing change in jenkins Config screen.
> My guess is easy enough to do... .
> ^^
> [~busbey] Lets chat (or I could bang my head)
> See HBASE-21424 for a bit of background.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21427) Make it so flakie's and nightlies build rates are configurable from jenkins Config page

2018-11-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675250#comment-16675250
 ] 

Sean Busbey commented on HBASE-21427:
-

IIRC it was not easy to do this because of how multi-branch pipelines work in 
jenkins.

What if we moved this stuff into its own repo, something like {{hbase-ci}}? we 
could probably remove most of {{dev-support}}. we'd need to refactor things a 
bit since the job would have to manage what branches are tested itself, but 
we'd gain the ability to throttle ourselves project wide as opposed to now 
where it's per-branch.

I went the per-branch route originally because it's what the jenkins docs 
encourage and it means we automagically have an equivalent amount of testing of 
new feature branches as the main branch. But I don't think folks are aware of 
the amount of testing done by default when they make a branch. Centralizing 
things would flip this around and make folks opt-in to it. We'd need to 
document that as a part of docs about "how to make a feature branch".

> Make it so flakie's and nightlies build rates are configurable from jenkins 
> Config page
> ---
>
> Key: HBASE-21427
> URL: https://issues.apache.org/jira/browse/HBASE-21427
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: stack
>Priority: Major
>
> Request is that rather than have to change code whenever we want to change 
> build rates, would be easier doing change in jenkins Config screen.
> My guess is easy enough to do... .
> ^^
> [~busbey] Lets chat (or I could bang my head)
> See HBASE-21424 for a bit of background.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21427) Make it so flakie's and nightlies build rates are configurable from jenkins Config page

2018-11-05 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21427:

Issue Type: Improvement  (was: Bug)

> Make it so flakie's and nightlies build rates are configurable from jenkins 
> Config page
> ---
>
> Key: HBASE-21427
> URL: https://issues.apache.org/jira/browse/HBASE-21427
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: stack
>Priority: Major
>
> Request is that rather than have to change code whenever we want to change 
> build rates, would be easier doing change in jenkins Config screen.
> My guess is easy enough to do... .
> ^^
> [~busbey] Lets chat (or I could bang my head)
> See HBASE-21424 for a bit of background.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21427) Make it so flakie's and nightlies build rates are configurable from jenkins Config page

2018-11-05 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21427:

Component/s: test

> Make it so flakie's and nightlies build rates are configurable from jenkins 
> Config page
> ---
>
> Key: HBASE-21427
> URL: https://issues.apache.org/jira/browse/HBASE-21427
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: stack
>Priority: Major
>
> Request is that rather than have to change code whenever we want to change 
> build rates, would be easier doing change in jenkins Config screen.
> My guess is easy enough to do... .
> ^^
> [~busbey] Lets chat (or I could bang my head)
> See HBASE-21424 for a bit of background.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?

2018-10-25 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664077#comment-16664077
 ] 

Sean Busbey commented on HBASE-21328:
-

Can you generate the patch using {{git format-patch}}?

> why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
> --
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation, Operability
>Reporter: Nick.han
>Assignee: Nick.han
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21328.master.001.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?

2018-10-25 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21328:

Fix Version/s: 3.0.0

> why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
> --
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation, Operability
>Reporter: Nick.han
>Assignee: Nick.han
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21328.master.001.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?

2018-10-25 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21328:

Component/s: Operability
 documentation

> why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
> --
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation, Operability
>Reporter: Nick.han
>Assignee: Nick.han
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21328.master.001.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?

2018-10-25 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reassigned HBASE-21328:
---

Assignee: Nick.han

> why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
> --
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation, Operability
>Reporter: Nick.han
>Assignee: Nick.han
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21328.master.001.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?

2018-10-25 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-21328:

Fix Version/s: 2.2.0
   1.5.0

> why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
> --
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation, Operability
>Reporter: Nick.han
>Assignee: Nick.han
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21328.master.001.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?

2018-10-25 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664070#comment-16664070
 ] 

Sean Busbey commented on HBASE-21328:
-

it's set by default to false to maintain existing behavior. calling it out in 
hbase-env seems like a reasonable incremental improvement.

> why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
> --
>
> Key: HBASE-21328
> URL: https://issues.apache.org/jira/browse/HBASE-21328
> Project: HBase
>  Issue Type: Improvement
>Reporter: Nick.han
>Priority: Minor
> Attachments: HBASE-21328.master.001.patch
>
>
> hi,all
>       I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to 
> build a hbase cluster,the problem is  hbase using javax.servlet-api-3.1.0-jar 
> witch is conflict by servlet-api-2.5.jar that in
> hadoop lib path, I run into hbase file and got config 
> HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config 
> decide whether or not include Hadoop lib to hbase class path,so the question 
> is why we set this config to false?can we set it to true and exclude the 
> Hadoop lib by default?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663150#comment-16663150
 ] 

Sean Busbey commented on HBASE-21215:
-

{code}
484   # Look for the -j /path/to/HBCK2.jar parameter. Else pass through to 
hbck.
485   case "${1}" in
486 -j*)
487 # Found -j parameter. Add arg to CLASSPATH and set CLASS to HBCK2.
488 shift
{code}

this should be
{code}
484   # Look for the -j /path/to/HBCK2.jar parameter. Else pass through to 
hbck.
485   case "${1}" in
486 -j)
487 # Found -j parameter. Add arg to CLASSPATH and set CLASS to HBCK2.
488 shift
{code}

specifically it's "-j)" instead of "-j*)"

otherwise you cover all possible options that start with "j' like
{code}
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck -jimminy-crickets 
~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar
 
usage: HBCK2 [OPTIONS] COMMAND 

Options:
 -d,--debug run with debug output
...
{code}

with that corrected I'm +1

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch, HBASE-21215.branch-2.1.003.patch, 
> HBASE-21215.branch-2.1.004.patch, HBASE-21215.branch-2.1.005.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662982#comment-16662982
 ] 

Sean Busbey commented on HBASE-21215:
-

{quote}
On --version giving NPE, I get the below. What is your context Sean Busbey sir?
{code}
$ ./bin/hbase hbck -j 
~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
 --version
project.version=1.0.0-SNAPSHOT
git.commit.id=7c5c3689b047614e34f7ac51de6a3cc5ccc84c1b
git.build.version=1.0.0-SNAPSHOT
{code}
{quote}

I built the hbck2 repo using {{-Dhbase.version=3.0.0-SNAPSHOT}} and then ran it 
against a local standalone cluster run out of the binary tarball created by 
master branch.

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch, HBASE-21215.branch-2.1.003.patch, 
> HBASE-21215.branch-2.1.004.patch, HBASE-21215.branch-2.1.005.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662933#comment-16662933
 ] 

Sean Busbey commented on HBASE-21215:
-

oh, my usage text is wrong. that would also need to be updated in the overall 
usage statement

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch, HBASE-21215.branch-2.1.003.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662926#comment-16662926
 ] 

Sean Busbey commented on HBASE-21215:
-

v3 is confusing still I think. arg handling still has different form then all 
the others in our scripts. handling home expansion for {{~}} doesn't cover any 
of the other shell expansion stuff.

This is what it would look like with plain {{--foo bar}} handling:

{code}
484  # Look for the --jar=/path/to/HBCK2.jar parameter. Else pass through to 
hbck.
485   case "${1}" in
486 --jar)
487 # Found --jar parameter. Add it to CLASSPATH and set CLASS to HBCK2.
488 .shift
488 JAR="${1}"
489 if [ ! -f "${JAR}" ]; then
490   echo "${JAR} file not found!"
491   echo "Usage: hbase [] hbck --jar=/path/to/HBCK2.jar 
[]"
492   exit 1
493 fi
...
{code}

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch, HBASE-21215.branch-2.1.003.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662901#comment-16662901
 ] 

Sean Busbey commented on HBASE-21215:
-

sorry for the back and forth. I looked at some more help output for our 
existing stuff and I think we should switch to the {{--foo bar}} handling. 
everything else does that AFAICT.

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662900#comment-16662900
 ] 

Sean Busbey commented on HBASE-21215:
-

presuming my reasoning above is correct

* logging on stdout is something to fix in hbck2
* {{--version}} gives NPE is something to fix in hbck2
* lack of shell expansion for paths isn't new to this and is something we can 
document instead of fixing
* ref guide update will have a follow-on jira

then I'm +1 on v2

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662898#comment-16662898
 ] 

Sean Busbey commented on HBASE-21215:
-

Default logging leaves a lot to be desired since the ZooKeeper noise is all 
mixed in with the program's expected output, but I think that's an HBCK2 thing 
and not an issue with this ticket:

{code}
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck --jar="$(echo 
~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar)"
 setTableState test ENABLED 2>/dev/null
16:39:58.361 [main] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to 
load native-hadoop library for your platform... using builtin-java classes 
where applicable
16:39:58.775 [main] INFO  org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient - 
Connect 0x687e99d8 to localhost:2181 with session timeout=9ms, retries 30, 
retry interval 1000ms, keepAlive=6ms
16:39:58.799 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, 
built on 03/23/2017 10:13 GMT
16:39:58.800 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:host.name=loaner-atx4968-mba
16:39:58.800 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:java.version=1.8.0_181
16:39:58.800 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:java.vendor=Oracle Corporation
16:39:58.800 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre
16:39:58.800 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:java.class.path=/Users/busbey/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../conf:/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home//lib/tools.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/..:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/shaded-clients/hbase-shaded-mapreduce-3.0.0-SNAPSHOT.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/log4j-1.2.17.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/shaded-clients/hbase-shaded-client-3.0.0-SNAPSHOT.jar:/Users/busbey/projects/hbase-project/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/bin/../lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar
16:39:58.801 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:java.library.path=/Users/busbey/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
16:39:58.801 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:java.io.tmpdir=/var/folders/n9/nmmnl94n343bqxpgvk7w98s0gn/T/
16:39:58.801 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:java.compiler=
16:39:58.801 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  
org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client 
environment:os.name=Mac OS X
16:39:58.801 [ReadOnlyZKClient-localhost:2181@0x687e99d8] INFO  

[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662889#comment-16662889
 ] 

Sean Busbey commented on HBASE-21215:
-

Presuming the below is an HBCK2 issue and not an issue with this patch, where 
do I file an issue for it?

{code}
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck --jar="$(echo 
~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar)"
 --version
Exception in thread "main" java.lang.NullPointerException
at java.io.Reader.(Reader.java:78)
at java.io.InputStreamReader.(InputStreamReader.java:72)
at org.apache.hbase.HBCK2.readHBCK2BuildProperties(HBCK2.java:210)
at org.apache.hbase.HBCK2.run(HBCK2.java:330)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hbase.HBCK2.main(HBCK2.java:426)
{code}

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662886#comment-16662886
 ] 

Sean Busbey edited comment on HBASE-21215 at 10/24/18 9:35 PM:
---

{code}
  # Look for the --jar=/path/to/HBCK2.jar parameter. Else pass through to hbck.
485   case "${1}" in
486 --jar=*)
487 # Found --jar parameter. Add it to CLASSPATH and set CLASS to HBCK2.
488 JAR="${1#*=}"
489 if [ ! -f "${JAR}" ]; then
490   echo "${JAR} file not found!"
491   echo "Usage: hbase [] hbck --jar=/path/to/HBCK2.jar 
[]"
492   exit 1
493 fi
{code}

because we use an argument handling of the form {{\-\-foo=bar}} instead of 
{{\-\-foo bar}}, we don't get shell expansion of the argument for things like 
{{~}}. e.g.
{code}
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck 
--jar=~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar
 
~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar
 file not found!
Usage: hbase [] hbck --jar=/path/to/HBCK2.jar []
{code}

probably a nit? the help does imply an absolute path. folks might find it 
confusing though.


was (Author: busbey):
{code}
  # Look for the --jar=/path/to/HBCK2.jar parameter. Else pass through to hbck.
485   case "${1}" in
486 --jar=*)
487 # Found --jar parameter. Add it to CLASSPATH and set CLASS to HBCK2.
488 JAR="${1#*=}"
489 if [ ! -f "${JAR}" ]; then
490   echo "${JAR} file not found!"
491   echo "Usage: hbase [] hbck --jar=/path/to/HBCK2.jar 
[]"
492   exit 1
493 fi
{code}

because we use an argument handling of the form {{--foo=bar}} instead of 
{{--foo bar}}, we don't get shell expansion of the argument for things like 
{{~}}. e.g.
{code}
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck 
--jar=~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar
 
~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar
 file not found!
Usage: hbase [] hbck --jar=/path/to/HBCK2.jar []
{code}

probably a nit? the help does imply an absolute path. folks might find it 
confusing though.

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662886#comment-16662886
 ] 

Sean Busbey commented on HBASE-21215:
-

{code}
  # Look for the --jar=/path/to/HBCK2.jar parameter. Else pass through to hbck.
485   case "${1}" in
486 --jar=*)
487 # Found --jar parameter. Add it to CLASSPATH and set CLASS to HBCK2.
488 JAR="${1#*=}"
489 if [ ! -f "${JAR}" ]; then
490   echo "${JAR} file not found!"
491   echo "Usage: hbase [] hbck --jar=/path/to/HBCK2.jar 
[]"
492   exit 1
493 fi
{code}

because we use an argument handling of the form {{--foo=bar}} instead of 
{{--foo bar}}, we don't get shell expansion of the argument for things like 
{{~}}. e.g.
{code}
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck 
--jar=~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar
 
~/.m2/repository/org/apache/hbase/hbase-hbck2/1.0.0-SNAPSHOT/hbase-hbck2-1.0.0-SNAPSHOT.jar
 file not found!
Usage: hbase [] hbck --jar=/path/to/HBCK2.jar []
{code}

probably a nit? the help does imply an absolute path. folks might find it 
confusing though.

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662878#comment-16662878
 ] 

Sean Busbey commented on HBASE-21215:
-

nit: error handling around the new arg has a gap. should get the usage warning 
here that's the same when the file doesn't exist:

{code}
loaner-atx4968-mba:hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck --jar  2>&1 | 
grep -v INFO
2018-10-24 16:26:53,178 WARN  [main] util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes where 
applicable
Unrecognized option:--jar

---
NOTE: As of HBase version 2.0, the hbck tool is significantly changed.
In general, all Read-Only options are supported and can be be used
safely. Most -fix/ -repair options are NOT supported. Please see usage
below for details on which options are not supported.
---

Usage: fsck [opts] {only tables}
 where [opts] are:
... rest of hbck 1 help...
{code}

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch, 
> HBASE-21215.branch-2.1.002.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662724#comment-16662724
 ] 

Sean Busbey commented on HBASE-21215:
-

@stack supposed to also update ref guide to truncate the hbck1 section and 
point folks over to the repo with hbck2.

{code}

+  ;;
+  *)
+  # Unknown param
+  echo "Unsupported parameter $SUBCOMMAND"
+  echo "Usage: hbase [] hbck --jar=/path/to/HBCK2.jar []"
+  exit 1
+  ;;
+esac
{code}

this is wrong I think? the jar parameter should be optional.

this fails as soon as you try telling the RO hbck1 something specific to look 
at. here are some examples that worked prior to the patch:

{code}
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck --help
Unsupported parameter --help
Usage: hbase [] hbck --jar=/path/to/HBCK2.jar []
hbase-3.0.0-SNAPSHOT busbey$ ./bin/hbase hbck -summary
Unsupported parameter -summary
Usage: hbase [] hbck --jar=/path/to/HBCK2.jar []
{code}


> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HBASE-21215) Figure how to invoke hbck2; make it easy to find

2018-10-24 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reassigned HBASE-21215:
---

Assignee: stack  (was: Sean Busbey)

> Figure how to invoke hbck2; make it easy to find
> 
>
> Key: HBASE-21215
> URL: https://issues.apache.org/jira/browse/HBASE-21215
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21215.branch-2.1.001.patch
>
>
> In 
> https://docs.google.com/document/d/1Oun4G3M5fyrM0OxXcCKYF8td0KD7gJQjnU9Ad-2t-uk/edit#,
>  the doc on hbck2 'form', one item to figure is how to invoke hbck2. Related, 
> how to make it easy to find? [~busbey] has some ideas (posted in doc). This 
> issue is for implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works

2018-10-24 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662653#comment-16662653
 ] 

Sean Busbey commented on HBASE-21381:
-

probably should be another blurb under the Hadoop matrix here:

[http://hbase.apache.org/book.html#hadoop]

 

bit awkward since B isn't in a release line.

 

maybe a call out in the B section instead? I can't think of another section 
that has hadoop specific details like this in it:

 

http://hbase.apache.org/book.html#backuprestore

> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
>  Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally 
> tried to concatenate the files being DistCp'ed to target cluster (though the 
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN  [Thread-936] mapred.LocalJobRunner$Job(590): 
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file 
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
>  length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry 
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
>  length = 5142 aclEntries = null, xAttrs = null}
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
>   at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so 
> that user of Backup and Restore feature knows which hadoop versions they can 
> use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HBASE-21380) TesRSGroups failing

2018-10-24 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reassigned HBASE-21380:
---

Assignee: Mike Drob

> TesRSGroups failing
> ---
>
> Key: HBASE-21380
> URL: https://issues.apache.org/jira/browse/HBASE-21380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Major
>
> only failing on branch-2.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-21380) TesRSGroups failing

2018-10-24 Thread Sean Busbey (JIRA)
Sean Busbey created HBASE-21380:
---

 Summary: TesRSGroups failing
 Key: HBASE-21380
 URL: https://issues.apache.org/jira/browse/HBASE-21380
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.1.1
Reporter: Sean Busbey


only failing on branch-2.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21371) Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license error

2018-10-23 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661139#comment-16661139
 ] 

Sean Busbey commented on HBASE-21371:
-

bq. Wei-Chiu Chuang - instead of listing cddl/gpl+ce as an acceptable license, 
I think we should update supplemental-models.xml to call out that we are 
specifically using the CDDL 1.1 half of that.

yes, this is definitely what we ought to do. Esp because the other license is 
not acceptable so we would not dual-license our redistribution. also CDDL 1.1 
is a license we aggregate (we include one copy and then list everything covered 
by it)

> Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license 
> error
> --
>
> Key: HBASE-21371
> URL: https://issues.apache.org/jira/browse/HBASE-21371
> Project: HBase
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-21371.001.patch
>
>
> Hadoop 3.3.0 (trunk) updated various dependencies, which adds additional 
> licenses that break HBase's license check plugin.
> CDDL/GPLv2+CE license
> {quote}This product includes JavaBeans Activation Framework API jar licensed 
> under the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
>  ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
>  javax.activation-api
>  1.2.0
> maven central search
>  g:javax.activation AND a:javax.activation-api AND v:1.2.0
> project website
>  [http://java.net/all/javax.activation-api/]
>  project source
>  [https://github.com/javaee/activation/javax.activation-api]
> {quote}
> Bouncy Castle License 
> {quote}–
>  This product includes Bouncy Castle PKIX, CMS, EAC, TSP, PKCS, OCSP, CMP, 
> and CRMF APIs licensed under the Bouncy Castle Licence.
> ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.bouncycastle
>  bcpkix-jdk15on
>  1.60
> maven central search
>  g:org.bouncycastle AND a:bcpkix-jdk15on AND v:1.60
> project website
>  [http://www.bouncycastle.org/java.html]
>  project source
>  [https://github.com/bcgit/bc-java]
>  –
> {quote}
>  
> And a long list of "Apache Software License - Version 2.0" licensed Jetty 
> dependencies like this:
> {quote}
> This product includes Jetty :: Servlet Annotations licensed under the Apache 
> Software License - Version 2.0.
> ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.eclipse.jetty
>  jetty-annotations
>  9.3.19.v20170502
> maven central search
>  g:org.eclipse.jetty AND a:jetty-annotations AND v:9.3.19.v20170502
> project website
>  [http://www.eclipse.org/jetty]
>  project source
>  [https://github.com/eclipse/jetty.project/jetty-annotations]
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21371) Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license error

2018-10-23 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661136#comment-16661136
 ] 

Sean Busbey commented on HBASE-21371:
-

bq. I should probably add a Bouncy Castle License text in LICENSE.vm as well. 
Though according to the project website, Bouncy Castle License is the same as 
MIT license.

yes, this will need to be done wherever we ship the relevant jar. if we already 
have bouncycastle as a dependency shouldn't we have an entry for it though?

> Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license 
> error
> --
>
> Key: HBASE-21371
> URL: https://issues.apache.org/jira/browse/HBASE-21371
> Project: HBase
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-21371.001.patch
>
>
> Hadoop 3.3.0 (trunk) updated various dependencies, which adds additional 
> licenses that break HBase's license check plugin.
> CDDL/GPLv2+CE license
> {quote}This product includes JavaBeans Activation Framework API jar licensed 
> under the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
>  ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
>  javax.activation-api
>  1.2.0
> maven central search
>  g:javax.activation AND a:javax.activation-api AND v:1.2.0
> project website
>  [http://java.net/all/javax.activation-api/]
>  project source
>  [https://github.com/javaee/activation/javax.activation-api]
> {quote}
> Bouncy Castle License 
> {quote}–
>  This product includes Bouncy Castle PKIX, CMS, EAC, TSP, PKCS, OCSP, CMP, 
> and CRMF APIs licensed under the Bouncy Castle Licence.
> ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.bouncycastle
>  bcpkix-jdk15on
>  1.60
> maven central search
>  g:org.bouncycastle AND a:bcpkix-jdk15on AND v:1.60
> project website
>  [http://www.bouncycastle.org/java.html]
>  project source
>  [https://github.com/bcgit/bc-java]
>  –
> {quote}
>  
> And a long list of "Apache Software License - Version 2.0" licensed Jetty 
> dependencies like this:
> {quote}
> This product includes Jetty :: Servlet Annotations licensed under the Apache 
> Software License - Version 2.0.
> ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.eclipse.jetty
>  jetty-annotations
>  9.3.19.v20170502
> maven central search
>  g:org.eclipse.jetty AND a:jetty-annotations AND v:9.3.19.v20170502
> project website
>  [http://www.eclipse.org/jetty]
>  project source
>  [https://github.com/eclipse/jetty.project/jetty-annotations]
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21371) Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license error

2018-10-23 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661135#comment-16661135
 ] 

Sean Busbey commented on HBASE-21371:
-

instead of adding yet another way of referring to ALv2 can we please update 
supplemental to correct the new phrasing?

> Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license 
> error
> --
>
> Key: HBASE-21371
> URL: https://issues.apache.org/jira/browse/HBASE-21371
> Project: HBase
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-21371.001.patch
>
>
> Hadoop 3.3.0 (trunk) updated various dependencies, which adds additional 
> licenses that break HBase's license check plugin.
> CDDL/GPLv2+CE license
> {quote}This product includes JavaBeans Activation Framework API jar licensed 
> under the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
>  ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
>  javax.activation-api
>  1.2.0
> maven central search
>  g:javax.activation AND a:javax.activation-api AND v:1.2.0
> project website
>  [http://java.net/all/javax.activation-api/]
>  project source
>  [https://github.com/javaee/activation/javax.activation-api]
> {quote}
> Bouncy Castle License 
> {quote}–
>  This product includes Bouncy Castle PKIX, CMS, EAC, TSP, PKCS, OCSP, CMP, 
> and CRMF APIs licensed under the Bouncy Castle Licence.
> ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.bouncycastle
>  bcpkix-jdk15on
>  1.60
> maven central search
>  g:org.bouncycastle AND a:bcpkix-jdk15on AND v:1.60
> project website
>  [http://www.bouncycastle.org/java.html]
>  project source
>  [https://github.com/bcgit/bc-java]
>  –
> {quote}
>  
> And a long list of "Apache Software License - Version 2.0" licensed Jetty 
> dependencies like this:
> {quote}
> This product includes Jetty :: Servlet Annotations licensed under the Apache 
> Software License - Version 2.0.
> ERROR: Please check  this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the 
> LICENSE.vm file.
>  If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.eclipse.jetty
>  jetty-annotations
>  9.3.19.v20170502
> maven central search
>  g:org.eclipse.jetty AND a:jetty-annotations AND v:9.3.19.v20170502
> project website
>  [http://www.eclipse.org/jetty]
>  project source
>  [https://github.com/eclipse/jetty.project/jetty-annotations]
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659222#comment-16659222
 ] 

Sean Busbey commented on HBASE-21247:
-

probably best to just save a copy of the class name for this.provider 
pre-wrapping, since if we need it for the meta wal we'll instantiate our own.

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v2.txt, 21247.v3.txt, 
> 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 21247.v6.txt, 21247.v7.txt, 
> 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659221#comment-16659221
 ] 

Sean Busbey commented on HBASE-21247:
-

oh even better, WALFactory is the one that does the wrapping in SRWP so we can 
just save a reference pre-wrapping

{code}
 if (enableSyncReplicationWALProvider) {
provider = new SyncReplicationWALProvider(provider);
  }

{code}

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v2.txt, 21247.v3.txt, 
> 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 21247.v6.txt, 21247.v7.txt, 
> 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659218#comment-16659218
 ] 

Sean Busbey commented on HBASE-21247:
-

we also probably don't want to use the SyncReplicationWALProvider as our Meta 
provider, since we don't want to replicate the meta wal via it (I presume?). I 
really wish replication would get out of our wal internals.

I suppose after we set up the main provider we'll know if it's the SRWP (and 
thus we need to pull the underlying WALProvider out)?

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v10.txt, 21247.v2.txt, 21247.v3.txt, 
> 21247.v4.tst, 21247.v4.txt, 21247.v5.txt, 21247.v6.txt, 21247.v7.txt, 
> 21247.v8.txt, 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-18792) hbase-2 needs to defend against hbck operations

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659205#comment-16659205
 ] 

Sean Busbey commented on HBASE-18792:
-

Could you folks move this conversation to the user@hbase mailing list? It'll 
get more eyes there.

> hbase-2 needs to defend against hbck operations
> ---
>
> Key: HBASE-18792
> URL: https://issues.apache.org/jira/browse/HBASE-18792
> Project: HBase
>  Issue Type: Task
>  Components: hbck
>Reporter: stack
>Assignee: Umesh Agashe
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-18792.branch-1.001.patch, 
> hbase-18792.master.001.patch, hbase-18792.master.002.patch
>
>
> hbck needs updating to run against hbase2. Meantime, if an hbck from hbase1 
> is run against hbck2, it may do damage. hbase2 should defend itself against 
> hbck1 ops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20448) update ref guide to expressly use shaded clients for examples

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659159#comment-16659159
 ] 

Sean Busbey commented on HBASE-20448:
-

I suspect the ITBLL issue is that what module we ship it in, since the 
classpath stuff is all geared towards being client-facing now and the ITBLL 
classes likely aren't included in the things a downstream MR application would 
need.

I'll work it out as a part of documenting things.

> update ref guide to expressly use shaded clients for examples
> -
>
> Key: HBASE-20448
> URL: https://issues.apache.org/jira/browse/HBASE-20448
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, documentation, mapreduce
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.2.0
>
>
> the whole mapreduce section, especially, should be using the shaded version.
> Specifically include an example of running the ITBLL we ship



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (HBASE-20331) clean up shaded packaging for 2.1

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-20331.
-
   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   (was: 3.0.0)
   2.1.0

with docs moved out all of the things in this umbrella were already done for 
2.1.0.

> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
>  Issue Type: Umbrella
>  Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more 
> contexts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20448) update ref guide to expressly use shaded clients for examples

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659157#comment-16659157
 ] 

Sean Busbey commented on HBASE-20448:
-

move to 2.2.0 and set to blocker

> update ref guide to expressly use shaded clients for examples
> -
>
> Key: HBASE-20448
> URL: https://issues.apache.org/jira/browse/HBASE-20448
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, documentation, mapreduce
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.2.0
>
>
> the whole mapreduce section, especially, should be using the shaded version.
> Specifically include an example of running the ITBLL we ship



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20448) update ref guide to expressly use shaded clients for examples

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20448:

Priority: Blocker  (was: Critical)

> update ref guide to expressly use shaded clients for examples
> -
>
> Key: HBASE-20448
> URL: https://issues.apache.org/jira/browse/HBASE-20448
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, documentation, mapreduce
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.2.0
>
>
> the whole mapreduce section, especially, should be using the shaded version.
> Specifically include an example of running the ITBLL we ship



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20448) update ref guide to expressly use shaded clients for examples

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20448:

Description: 
the whole mapreduce section, especially, should be using the shaded version.

Specifically include an example of running the ITBLL we ship

  was:the whole mapreduce section, especially, should be using the shaded 
version.


> update ref guide to expressly use shaded clients for examples
> -
>
> Key: HBASE-20448
> URL: https://issues.apache.org/jira/browse/HBASE-20448
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, documentation, mapreduce
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.2.0
>
>
> the whole mapreduce section, especially, should be using the shaded version.
> Specifically include an example of running the ITBLL we ship



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20448) update ref guide to expressly use shaded clients for examples

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20448:

Issue Type: Improvement  (was: Sub-task)
Parent: (was: HBASE-20331)

> update ref guide to expressly use shaded clients for examples
> -
>
> Key: HBASE-20448
> URL: https://issues.apache.org/jira/browse/HBASE-20448
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, documentation, mapreduce
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.2.0
>
>
> the whole mapreduce section, especially, should be using the shaded version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Work stopped] (HBASE-20448) update ref guide to expressly use shaded clients for examples

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-20448 stopped by Sean Busbey.
---
> update ref guide to expressly use shaded clients for examples
> -
>
> Key: HBASE-20448
> URL: https://issues.apache.org/jira/browse/HBASE-20448
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, documentation, mapreduce
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.2.0
>
>
> the whole mapreduce section, especially, should be using the shaded version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659155#comment-16659155
 ] 

Sean Busbey commented on HBASE-20331:
-

docs wasn't as easy as I thought. classic story. Let me move the docs target to 
2.2 and close this out

> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
>  Issue Type: Umbrella
>  Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more 
> contexts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20448) update ref guide to expressly use shaded clients for examples

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20448:

Fix Version/s: (was: 2.1.1)
   2.2.0

> update ref guide to expressly use shaded clients for examples
> -
>
> Key: HBASE-20448
> URL: https://issues.apache.org/jira/browse/HBASE-20448
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, documentation, mapreduce
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.2.0
>
>
> the whole mapreduce section, especially, should be using the shaded version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20331) clean up shaded packaging for 2.1

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20331:

Fix Version/s: (was: 2.1.1)

> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
>  Issue Type: Umbrella
>  Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more 
> contexts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659153#comment-16659153
 ] 

Sean Busbey commented on HBASE-21247:
-

Catching a {{Throwable}} is v dangerous and IMHO not justified for a code path 
that is essentially just handling config fallback. Also why do another lookup 
on WAL_PROVIDER instead of just passing in DEFAULT_WAL_PROVIDER?

With the shift to taking a {{Class}} for the default 
then the call in {{getMetaProvider}} can just look like this:

{code}
provider = createProvider(getProviderClass(META_WAL_PROVIDER, 
this.provider.class));
{code}

This makes it very clear that should META_WAL_PROVIDER be undefined we'll use 
the same class as the non-meta wal provider. It also coincidentally avoids 
doing repeated look ups on the WAL_PROVIDER config.

We can remove the {{DEFAULT_WAL_PROVIDER}} constant entirely and update the 
WALFactory constructor that creates {{this.provider}} to use 
{{Providers.defaultProvider.clazz}} directly.

Much nicer, no?

> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 
> 21247.v4.txt, 21247.v5.txt, 21247.v6.txt, 21247.v7.txt, 21247.v8.txt, 
> 21247.v9.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20105) Allow flushes to target SSD storage

2018-10-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20105:

Status: In Progress  (was: Patch Available)

moving out of patch available pending update based on review feedback.

> Allow flushes to target SSD storage
> ---
>
> Key: HBASE-20105
> URL: https://issues.apache.org/jira/browse/HBASE-20105
> Project: HBase
>  Issue Type: New Feature
>  Components: Performance, regionserver
>Affects Versions: hbase-2.0.0-alpha-4
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20105-v0.patch, HBASE-20105-v1.patch, 
> HBASE-20105-v2.patch, HBASE-20105-v3.patch, HBASE-20105-v4.patch, 
> HBASE-20105-v5.patch, HBASE-20105-v6.patch
>
>
> On heavy writes usecases, flushes are compactes together pretty quickly. 
> Allowing flushes to go on SSD allows faster flush and faster first 
> compactions. Subsequent compactions going on regular storage.
>  
> I will be interesting to have an option to target SSD for flushes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers

2018-10-22 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658950#comment-16658950
 ] 

Sean Busbey commented on HBASE-21247:
-

okay I've made a first pass. I like the added test and the correction on 
IOTestProvider.

I agree that your current solution solves the immediate issue, but I think it's 
confusing to follow and will be hard to maintain. In particular:

{code}

   // Fall back to them specifying a class name
   // Note that the passed default class shouldn't actually be used, since 
the above only fails
   // when there is a config value present.
-  return conf.getClass(key, Providers.defaultProvider.clazz, 
WALProvider.class);
+  // If meta WAL provider is not specified, use provider class of ordinary 
WAL
+  return conf.getClass(key, key.equals(META_WAL_PROVIDER) && 
conf.get(META_WAL_PROVIDER)==null ?
+  conf.getClass(WAL_PROVIDER, Providers.defaultProvider.clazz, 
WALProvider.class) :
+Providers.defaultProvider.clazz,
+  WALProvider.class);
{code}

It's odd to suddenly read a bunch of stuff about the specific provider keys at 
the end of a method that was generic and reusable before then. Additionally, 
the comment says "we won't use this default because of XXX" and then we do a 
bunch of logic to determine what that default ought to be.

I think the fundamental issue is that the way HBASE-20856 was implemented 
invalidated the assumptions of this method; specifically that the default 
passed to {{getProviderClass}} would be a member of the Providers enum. I think 
it'll be easier to maintain this long term if we either restore that assumption 
or make it no longer needed, rather than do a second round of configuration 
lookups in this exception handling block.

What if we refactor {{getProviderClass}} to take a {{Class}} as the 
defaultValue rather than a string version? We can get a String out of it for 
the call to {{Providers.valueOf}} and then use it directly if needed when we 
call {{Configuration.getClass}}.



> Custom WAL Provider cannot be specified by configuration whose value is 
> outside the enums in Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 
> 21247.v4.txt, 21247.v5.txt, 21247.v6.txt, 21247.v7.txt, 21247.v8.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue fixes the bug by allowing the specification of new WAL Provider 
> class name using the config "hbase.wal.provider".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (HBASE-21302) Release 1.2.8

2018-10-21 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-21302.
-
Resolution: Fixed

release announcement sent to announce@apache and {dev,user}@hbase

> Release 1.2.8
> -
>
> Key: HBASE-21302
> URL: https://issues.apache.org/jira/browse/HBASE-21302
> Project: HBase
>  Issue Type: Task
>  Components: community
>Affects Versions: 1.2.8
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 1.2.8
>
>
> 1.4.8 is out, time to make 1.2.8.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


  1   2   3   4   5   6   7   8   9   10   >