[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295969#comment-16295969
]
Duo Zhang commented on HBASE-14790:
---
{quote}
any interest in contributing this to HDFS?
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295639#comment-16295639
]
Arpit Agarwal commented on HBASE-14790:
---
This looks interesting.
[~Apache9], any i
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295235#comment-16295235
]
stack commented on HBASE-14790:
---
Thats a great RN.
> Implement a new DFSOutputStream for l
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293669#comment-16293669
]
Duo Zhang commented on HBASE-14790:
---
Let me do it. Thanks the reminder sir [~stack].
>
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292812#comment-16292812
]
stack commented on HBASE-14790:
---
Needs a fat release note [~Apache9]
> Implement a new DFS
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292708#comment-16292708
]
stack commented on HBASE-14790:
---
Wahoo!
> Implement a new DFSOutputStream for logging WAL
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145757#comment-15145757
]
Duo Zhang commented on HBASE-14790:
---
{quote}
E.g. gen stamp bumping, token renewal.
{q
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145631#comment-15145631
]
Zhe Zhang commented on HBASE-14790:
---
Nice work here [~Apache9]. By quickly browsing thr
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15137286#comment-15137286
]
Yu Li commented on HBASE-14790:
---
bq. Yeah we really need something between hflush and hsync
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136991#comment-15136991
]
Elliott Clark commented on HBASE-14790:
---
I was looking at WALProcedure rather than
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136985#comment-15136985
]
Duo Zhang commented on HBASE-14790:
---
[~eclark] This is the implementation of {{Protobuf
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136961#comment-15136961
]
Elliott Clark commented on HBASE-14790:
---
For any data that HBase will ever consider
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135606#comment-15135606
]
Duo Zhang commented on HBASE-14790:
---
Fine. Let's do it in HBase.
A little problem is t
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135542#comment-15135542
]
stack commented on HBASE-14790:
---
First, what you think of the 2x number? I think it is kind
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135501#comment-15135501
]
Duo Zhang commented on HBASE-14790:
---
[~stack] Oh there is a WALPE tool, I didn't know i
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135485#comment-15135485
]
Duo Zhang commented on HBASE-14790:
---
[~eclark] AFAIK, the ack of hflush only means that
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135272#comment-15135272
]
Elliott Clark commented on HBASE-14790:
---
bq.or three DNs and RS crash at the same t
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135250#comment-15135250
]
stack commented on HBASE-14790:
---
I gave it a try on a little cluster. Not all the metrics a
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133896#comment-15133896
]
Duo Zhang commented on HBASE-14790:
---
I have got a 5 regionservers test cluster and run
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111963#comment-15111963
]
Duo Zhang commented on HBASE-14790:
---
https://github.com/Apache9/hbase/blob/HBASE-14790/
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074735#comment-15074735
]
Duo Zhang commented on HBASE-14790:
---
{quote}
Soon as you have a basic WAL Duo Zhang, I
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074678#comment-15074678
]
stack commented on HBASE-14790:
---
Soon as you have a basic WAL [~Apache9], I can try get som
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074641#comment-15074641
]
Duo Zhang commented on HBASE-14790:
---
https://github.com/Apache9/hbase/blob/HBASE-14790/
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042639#comment-15042639
]
Duo Zhang commented on HBASE-14790:
---
We just dig into the implementation of {{DFSOutput
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042228#comment-15042228
]
Zhe Zhang commented on HBASE-14790:
---
Thanks for the catch. If needed I guess we can mak
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041910#comment-15041910
]
Phil Yang commented on HBASE-14790:
---
Currently there are two scenarios which may result
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041761#comment-15041761
]
Yu Li commented on HBASE-14790:
---
bq. hsync periodically, for example, default per 1000ms
AF
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041321#comment-15041321
]
Duo Zhang commented on HBASE-14790:
---
{quote}
This is clean up of a broken WAL? This is
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041307#comment-15041307
]
Heng Chen commented on HBASE-14790:
---
{quote}
hsync periodically, for example, default p
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041234#comment-15041234
]
Phil Yang commented on HBASE-14790:
---
Considering these features:
Hflush is much faster
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041196#comment-15041196
]
Heng Chen commented on HBASE-14790:
---
{quote}
DataStreamer#block tracks the "number of b
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041185#comment-15041185
]
stack commented on HBASE-14790:
---
bq. ReplicationSource should ask this length first before
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041147#comment-15041147
]
Duo Zhang commented on HBASE-14790:
---
And I found that, {{hsync}} and {{hflush}} have di
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041145#comment-15041145
]
Heng Chen commented on HBASE-14790:
---
Make sense...
Let's just keep here as original.
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15040809#comment-15040809
]
Duo Zhang commented on HBASE-14790:
---
[~chenheng] We should make a trade off here. I do
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15040781#comment-15040781
]
Heng Chen commented on HBASE-14790:
---
{quote}
And for keeping an "acked length", I think
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039770#comment-15039770
]
Duo Zhang commented on HBASE-14790:
---
https://issues.apache.org/jira/secure/attachment/1
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039718#comment-15039718
]
Duo Zhang commented on HBASE-14790:
---
{quote}
2. dn1 received the WAL entry, and it is r
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038849#comment-15038849
]
Duo Zhang commented on HBASE-14790:
---
[~zhz] I was wrong,the pipeline recovery is always
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038259#comment-15038259
]
stack commented on HBASE-14790:
---
[~zhz]
bq. DataStreamer#block tracks the "number of bytes
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038178#comment-15038178
]
Zhe Zhang commented on HBASE-14790:
---
[~stack] {{DataStreamer#block}} tracks the "number
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038167#comment-15038167
]
stack commented on HBASE-14790:
---
bq. So I think every rs should keep an "acked length" of t
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037630#comment-15037630
]
Yu Li commented on HBASE-14790:
---
Agree that we may not fix HBASE-14004 by simply implementi
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037472#comment-15037472
]
Duo Zhang commented on HBASE-14790:
---
Oh, I think we could not fix HBASE-14004 without c
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037338#comment-15037338
]
Duo Zhang commented on HBASE-14790:
---
I read the code in {{NameNode}} and {{DFSOutputStr
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035230#comment-15035230
]
Duo Zhang commented on HBASE-14790:
---
basically works. I added a test for it
https://gi
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034663#comment-15034663
]
Zhe Zhang commented on HBASE-14790:
---
Thanks for the clarification Duo.
bq. Here I'm tr
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15033281#comment-15033281
]
Duo Zhang commented on HBASE-14790:
---
Thanks for the information.
Here I'm trying to us
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15033245#comment-15033245
]
Zhe Zhang commented on HBASE-14790:
---
On the high level, the key requirements here are v
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15031479#comment-15031479
]
Duo Zhang commented on HBASE-14790:
---
Now I‘m trying to implement a fan-out async output
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005449#comment-15005449
]
Heng Chen commented on HBASE-14790:
---
IMO we could fix HBASE-14004 as the first step.
B
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003638#comment-15003638
]
stack commented on HBASE-14790:
---
[~wheat9]
bq. The potential issue I see is that the DN mig
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003636#comment-15003636
]
stack commented on HBASE-14790:
---
[~busbey]
bq. That's old code I've yet to see benchmark j
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003595#comment-15003595
]
Duo Zhang commented on HBASE-14790:
---
HTTP/2 has its own problems that we haven't finish
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003560#comment-15003560
]
Haohui Mai commented on HBASE-14790:
Making the errors in the pipeline visible to HBa
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003477#comment-15003477
]
Sean Busbey commented on HBASE-14790:
-
We don't need to sync in a different thread. T
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003469#comment-15003469
]
Duo Zhang commented on HBASE-14790:
---
The new implementation will be event-driven which
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003450#comment-15003450
]
Enis Soztutar commented on HBASE-14790:
---
Sounds like a good idea to not have the pi
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001665#comment-15001665
]
Yu Li commented on HBASE-14790:
---
Thanks for the explanation [~Apache9] and [~busbey]. So an
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001319#comment-15001319
]
Sean Busbey commented on HBASE-14790:
-
Other differences from when I was looking:
*
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1417#comment-1417
]
Duo Zhang commented on HBASE-14790:
---
{{DFSOutputStream}} has pipeline recovery, so it i
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1410#comment-1410
]
Yu Li commented on HBASE-14790:
---
[~Apache9]
{quote}
So a simple solution is, when logging f
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999330#comment-14999330
]
Sean Busbey commented on HBASE-14790:
-
excellent! I had a first pass of this done sev
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998407#comment-14998407
]
Duo Zhang commented on HBASE-14790:
---
{quote}
As for this, IMO we can set a limit, if ex
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998403#comment-14998403
]
Heng Chen commented on HBASE-14790:
---
{quote}
try closing the file(just make a call to
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998397#comment-14998397
]
Duo Zhang commented on HBASE-14790:
---
The root reason of HBASE-14004 is that, HBase and
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998312#comment-14998312
]
Heng Chen commented on HBASE-14790:
---
{quote}
And the most important thing is that, it i
67 matches
Mail list logo