[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295969#comment-16295969
]
Duo Zhang commented on HBASE-14790:
---
{quote}
any interest in contributing this to HDFS?
{quote}
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295639#comment-16295639
]
Arpit Agarwal commented on HBASE-14790:
---
This looks interesting.
[~Apache9], any interest in
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295235#comment-16295235
]
stack commented on HBASE-14790:
---
Thats a great RN.
> Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293669#comment-16293669
]
Duo Zhang commented on HBASE-14790:
---
Let me do it. Thanks the reminder sir [~stack].
> Implement a new
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292812#comment-16292812
]
stack commented on HBASE-14790:
---
Needs a fat release note [~Apache9]
> Implement a new DFSOutputStream for
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292708#comment-16292708
]
stack commented on HBASE-14790:
---
Wahoo!
> Implement a new DFSOutputStream for logging WAL only
>
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145631#comment-15145631
]
Zhe Zhang commented on HBASE-14790:
---
Nice work here [~Apache9]. By quickly browsing through the code, I
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145757#comment-15145757
]
Duo Zhang commented on HBASE-14790:
---
{quote}
E.g. gen stamp bumping, token renewal.
{quote}
Yes, this
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136961#comment-15136961
]
Elliott Clark commented on HBASE-14790:
---
For any data that HBase will ever consider to be durable
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136985#comment-15136985
]
Duo Zhang commented on HBASE-14790:
---
[~eclark] This is the implementation of {{ProtobufLogWriter.sync}}
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136991#comment-15136991
]
Elliott Clark commented on HBASE-14790:
---
I was looking at WALProcedure rather than protobuf.
Yeah
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137286#comment-15137286
]
Yu Li commented on HBASE-14790:
---
bq. Yeah we really need something between hflush and hsync. The current
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135250#comment-15135250
]
stack commented on HBASE-14790:
---
I gave it a try on a little cluster. Not all the metrics are hooked up and
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133896#comment-15133896
]
Duo Zhang commented on HBASE-14790:
---
I have got a 5 regionservers test cluster and run pe tool with
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135501#comment-15135501
]
Duo Zhang commented on HBASE-14790:
---
[~stack] Oh there is a WALPE tool, I didn't know it before, I have
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135485#comment-15135485
]
Duo Zhang commented on HBASE-14790:
---
[~eclark] AFAIK, the ack of hflush only means that datanode has
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135542#comment-15135542
]
stack commented on HBASE-14790:
---
First, what you think of the 2x number? I think it is kinda cool.
On #1,
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135272#comment-15135272
]
Elliott Clark commented on HBASE-14790:
---
bq.or three DNs and RS crash at the same time
This isn't
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135606#comment-15135606
]
Duo Zhang commented on HBASE-14790:
---
Fine. Let's do it in HBase.
A little problem is that if we get an
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111963#comment-15111963
]
Duo Zhang commented on HBASE-14790:
---
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074641#comment-15074641
]
Duo Zhang commented on HBASE-14790:
---
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074678#comment-15074678
]
stack commented on HBASE-14790:
---
Soon as you have a basic WAL [~Apache9], I can try get some basic numbers?
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074735#comment-15074735
]
Duo Zhang commented on HBASE-14790:
---
{quote}
Soon as you have a basic WAL Duo Zhang, I can try get some
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041307#comment-15041307
]
Heng Chen commented on HBASE-14790:
---
{quote}
hsync periodically, for example, default per 1000ms? It
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041321#comment-15041321
]
Duo Zhang commented on HBASE-14790:
---
{quote}
This is clean up of a broken WAL? This is being able to
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041761#comment-15041761
]
Yu Li commented on HBASE-14790:
---
bq. hsync periodically, for example, default per 1000ms
AFAIK, DN will do
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042228#comment-15042228
]
Zhe Zhang commented on HBASE-14790:
---
Thanks for the catch. If needed I guess we can make it public. Or
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042639#comment-15042639
]
Duo Zhang commented on HBASE-14790:
---
We just dig into the implementation of {{DFSOutputStream}} and
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041910#comment-15041910
]
Phil Yang commented on HBASE-14790:
---
Currently there are two scenarios which may result in
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041145#comment-15041145
]
Heng Chen commented on HBASE-14790:
---
Make sense...
Let's just keep here as original. We can only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041196#comment-15041196
]
Heng Chen commented on HBASE-14790:
---
{quote}
DataStreamer#block tracks the "number of bytes acked". It
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039718#comment-15039718
]
Duo Zhang commented on HBASE-14790:
---
{quote}
2. dn1 received the WAL entry, and it is read by
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041185#comment-15041185
]
stack commented on HBASE-14790:
---
bq. ReplicationSource should ask this length first before reading and do
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15040781#comment-15040781
]
Heng Chen commented on HBASE-14790:
---
{quote}
And for keeping an "acked length", I think we could make
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041234#comment-15041234
]
Phil Yang commented on HBASE-14790:
---
Considering these features:
Hflush is much faster than hsync,
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15040809#comment-15040809
]
Duo Zhang commented on HBASE-14790:
---
[~chenheng] We should make a trade off here. I do not think
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037472#comment-15037472
]
Duo Zhang commented on HBASE-14790:
---
Oh, I think we could not fix HBASE-14004 without changing the
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037630#comment-15037630
]
Yu Li commented on HBASE-14790:
---
Agree that we may not fix HBASE-14004 by simply implementing a new
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039770#comment-15039770
]
Duo Zhang commented on HBASE-14790:
---
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041147#comment-15041147
]
Duo Zhang commented on HBASE-14790:
---
And I found that, {{hsync}} and {{hflush}} have different ack
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038167#comment-15038167
]
stack commented on HBASE-14790:
---
bq. So I think every rs should keep an "acked length" of the current
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038178#comment-15038178
]
Zhe Zhang commented on HBASE-14790:
---
[~stack] {{DataStreamer#block}} tracks the "number of bytes
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038849#comment-15038849
]
Duo Zhang commented on HBASE-14790:
---
[~zhz] I was wrong，the pipeline recovery is always needed in the
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038259#comment-15038259
]
stack commented on HBASE-14790:
---
[~zhz]
bq. DataStreamer#block tracks the "number of bytes acked". It is
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037338#comment-15037338
]
Duo Zhang commented on HBASE-14790:
---
I read the code in {{NameNode}} and {{DFSOutputStream}} and I
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034663#comment-15034663
]
Zhe Zhang commented on HBASE-14790:
---
Thanks for the clarification Duo.
bq. Here I'm trying to use only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035230#comment-15035230
]
Duo Zhang commented on HBASE-14790:
---
basically works. I added a test for it
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033245#comment-15033245
]
Zhe Zhang commented on HBASE-14790:
---
On the high level, the key requirements here are very similar to
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033281#comment-15033281
]
Duo Zhang commented on HBASE-14790:
---
Thanks for the information.
Here I'm trying to use only one
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031479#comment-15031479
]
Duo Zhang commented on HBASE-14790:
---
Now I‘m trying to implement a fan-out async output stream first.
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005449#comment-15005449
]
Heng Chen commented on HBASE-14790:
---
IMO we could fix HBASE-14004 as the first step.
Before we
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003477#comment-15003477
]
Sean Busbey commented on HBASE-14790:
-
We don't need to sync in a different thread. That's old code
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003638#comment-15003638
]
stack commented on HBASE-14790:
---
[~wheat9]
bq. The potential issue I see is that the DN might mask the
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003560#comment-15003560
]
Haohui Mai commented on HBASE-14790:
Making the errors in the pipeline visible to HBase allows HBase
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003595#comment-15003595
]
Duo Zhang commented on HBASE-14790:
---
HTTP/2 has its own problems that we haven't finish the read path
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003636#comment-15003636
]
stack commented on HBASE-14790:
---
[~busbey]
bq. That's old code I've yet to see benchmark justification
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003450#comment-15003450
]
Enis Soztutar commented on HBASE-14790:
---
Sounds like a good idea to not have the pipeline recovery
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003469#comment-15003469
]
Duo Zhang commented on HBASE-14790:
---
The new implementation will be event-driven which means we could
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001665#comment-15001665
]
Yu Li commented on HBASE-14790:
---
Thanks for the explanation [~Apache9] and [~busbey]. So an optimized
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001319#comment-15001319
]
Sean Busbey commented on HBASE-14790:
-
Other differences from when I was looking:
* current
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999330#comment-14999330
]
Sean Busbey commented on HBASE-14790:
-
excellent! I had a first pass of this done several months ago
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998312#comment-14998312
]
Heng Chen commented on HBASE-14790:
---
{quote}
And the most important thing is that, it is hard to handle
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998397#comment-14998397
]
Duo Zhang commented on HBASE-14790:
---
The root reason of HBASE-14004 is that, HBase and HDFS may not
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998403#comment-14998403
]
Heng Chen commented on HBASE-14790:
---
{quote}
try closing the file(just make a call to namenode with
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998407#comment-14998407
]
Duo Zhang commented on HBASE-14790:
---
{quote}
As for this, IMO we can set a limit, if exceed the limits,
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1410#comment-1410
]
Yu Li commented on HBASE-14790:
---
[~Apache9]
{quote}
So a simple solution is, when logging failed, try
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1417#comment-1417
]
Duo Zhang commented on HBASE-14790:
---
{{DFSOutputStream}} has pipeline recovery, so it is hard to say
67 matches
Mail list logo
