[jira] [Created] (HBASE-16684) The get() requests does not see locally buffered put() requests when autoflush is disabled
Haohui Mai created HBASE-16684: -- Summary: The get() requests does not see locally buffered put() requests when autoflush is disabled Key: HBASE-16684 URL: https://issues.apache.org/jira/browse/HBASE-16684 Project: HBase Issue Type: Improvement Reporter: Haohui Mai Priority: Minor When autoflush is disabled the put() requests are buffered locally. Subsequent get() requests on the same host will always go to the network and will not see the updates that are buffered locally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003560#comment-15003560
]
Haohui Mai commented on HBASE-14790:
Making the errors in the pipeline visible to HBase allows HBase to detect
failures and to recover from failures much faster. It has a lot of benefits in
terms on reducing the latency of HBase.
An Exokernel style writer will eventually allow HBase to write to HDFS in
parallel, which further reducing the latency by 3x.
I would suggest (1) implementing the writer in the HDFS project to reduce the
cost of maintenance, (2) making it event-driven so that it is reusable when
building today's {{DFSOutputStream}}. It's much harder to do so today as there
are a lot of synchronization happening for throttling, etc.
It is relatively straightforward to implement the current client-side, pipeline
protocol without handling failures. The potential issue I see is that the DN
might mask the failures and introduce additional delays in the pipeline. To
fully get the benefits it might require changing the protocol. That's being
said, the project suddenly becomes much risker when it requires changes on the
server side.
A less risky route is to combine the effort with the HTTP/2 initiatives of HDFS
which allows full control on both the client and the server side. Thoughts?
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879256#comment-13879256
]
Haohui Mai commented on HBASE-10336:
As [~sureshms] suggested in HADOOP-10253, an alternative approach is to
maintain a temporary copy of {{HttpServer}} in hadoop-common for HBase in 2.4.
That way hbase can work with hadoop 2.4 untile HBASE-10336 is landed.
> Remove deprecated usage of Hadoop HttpServer in InfoServer
> --
>
> Key: HBASE-10336
> URL: https://issues.apache.org/jira/browse/HBASE-10336
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Eric Charles
>Assignee: Eric Charles
> Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch,
> HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch
>
>
> Recent changes in Hadoop HttpServer give NPE when running on hadoop
> 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be
> not fixed (see HDFS-5760). We'd better move to the new proposed builder
> pattern, which means we can no more use inheritance to build our nice
> InfoServer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879037#comment-13879037
]
Haohui Mai commented on HBASE-10336:
HDFS-5305 and related jiras implement HTTPS support, as a side effect they have
changed a lot how {{HttpServer}} works.
I anticipate that there are more changes for {{HttpServer}} in the near term.
We want to keep it as an private API so that we can keep cleaning it up and
without worrying about compatibility.
For distribution, the approach of this jira might work better. It seems to me
that this is a reliable way to not making the table you've mentioned more
frightening. :-)
> Remove deprecated usage of Hadoop HttpServer in InfoServer
> --
>
> Key: HBASE-10336
> URL: https://issues.apache.org/jira/browse/HBASE-10336
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Eric Charles
>Assignee: Eric Charles
> Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch,
> HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch
>
>
> Recent changes in Hadoop HttpServer give NPE when running on hadoop
> 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be
> not fixed (see HDFS-5760). We'd better move to the new proposed builder
> pattern, which means we can no more use inheritance to build our nice
> InfoServer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878979#comment-13878979
]
Haohui Mai commented on HBASE-10336:
The following code has been deprecated:
{code}
+
+ static {
+Configuration conf = new Configuration();
+boolean sslEnabled = conf.getBoolean(
+CommonConfigurationKeysPublic.HADOOP_SSL_ENABLED_KEY,
+CommonConfigurationKeysPublic.HADOOP_SSL_ENABLED_DEFAULT);
+policy = sslEnabled ? Policy.HTTPS_ONLY : Policy.HTTP_ONLY;
+ }
+
+ public static void setPolicy(Policy policy) {
+HttpConfig.policy = policy;
+ }
+
+ public static boolean isSecure() {
+return policy == Policy.HTTPS_ONLY;
+ }
+
+ public static String getSchemePrefix() {
+return (isSecure()) ? "https://"; : "http://";;
+ }
+
+ public static String getScheme(Policy policy) {
+return policy == Policy.HTTPS_ONLY ? "https://"; : "http://";;
+ }
{code}
{code}
+/**
+ * Use setAppDir() instead.
+ */
+@Deprecated
+public Builder setName(String name){
+ this.name = name;
+ return this;
+}
+
+/**
+ * Use addEndpoint() instead.
+ */
+@Deprecated
+public Builder setBindAddress(String bindAddress){
+ this.bindAddress = bindAddress;
+ return this;
+}
+
+/**
+ * Use addEndpoint() instead.
+ */
+@Deprecated
+public Builder setPort(int port) {
+ this.port = port;
+ return this;
+}
+
{code}
And there are significant amount of code in {{HttpServer}} has been marked as
deprecated. You might want to get rid of them as well.
> Remove deprecated usage of Hadoop HttpServer in InfoServer
> --
>
> Key: HBASE-10336
> URL: https://issues.apache.org/jira/browse/HBASE-10336
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Eric Charles
>Assignee: Eric Charles
> Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch,
> HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch
>
>
> Recent changes in Hadoop HttpServer give NPE when running on hadoop
> 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be
> not fixed (see HDFS-5760). We'd better move to the new proposed builder
> pattern, which means we can no more use inheritance to build our nice
> InfoServer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
