[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301244#comment-16301244 ] Niels Basjes commented on HBASE-19486: -- Ok, thanks for clarifying. I'll get on it in the next few days. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch, > HBASE-19486-20171220-2228-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299407#comment-16299407
]
Chia-Ping Tsai commented on HBASE-19486:
Seems {{autoFlushCount}] is used by test only? If so, could we make
{{autoFlush()}} be protected and then overrride it in test to get the count of
auto flush.
Should we cancel the previous task before invoking new timer task?
{code}
+ autoFlushTimer = new Timer(true); // Create Timer running as Daemon.
+ autoFlushTimer.schedule(new TimerTask() {
+@Override
+public void run() {
+ BufferedMutatorImpl.this.autoFlush();
+}
+ }, writeBufferMaxLingerMs, writeBufferMaxLingerMs);
{code}
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch,
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299334#comment-16299334 ] stack commented on HBASE-19486: --- What [~jerryhe] said. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch, > HBASE-19486-20171220-2228-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299280#comment-16299280
]
Jerry He commented on HBASE-19486:
--
I try to get a better feel for the word 'linger' to see if it fits better here.
Still like {{write buffer periodic flush interva}} better at the moment to fit
into hbase.
Thanks for the effort to explain it.
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch,
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299135#comment-16299135
]
Niels Basjes commented on HBASE-19486:
--
[~stack] Ok, I understand {{autoflush}} should not be used.
Current name of the setting is {{writeBufferMaxLingerMs}}
The only thing I currently named {{autoflush}} is the timer. I realize that
this is inconsistent with the rest of this change (renaming that to
{{writeBufferMaxLingerTimer}} is easy).
But before I do this: What naming for this feature would you guys prefer?
Something like {{write buffer periodic flush interval}} perhaps?
Or is just getting rid of {{autoflush}} what you want?
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch,
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299076#comment-16299076 ] stack commented on HBASE-19486: --- RN is lovely. Appreciate your trying to align us w/ Kafka. Good. But [~jerryhe] has a point. autoflush in a client context particularly around BufferedMutator meant something else... See http://hbase.apache.org/1.2/apidocs/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush-boolean- Probably better to name it something else sir. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299034#comment-16299034 ] Niels Basjes commented on HBASE-19486: -- [~stack] I added a first draft release note to this issue. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299017#comment-16299017
]
Niels Basjes commented on HBASE-19486:
--
@jinghe I used similar naming from what I know from Kafka which has a setting
which effectively does something very similar:
Look for {{linger.ms}} in this page: https://kafka.apache.org/documentation/
I personally think that having a similar naming compared to Kafka for this
makes it easier for "Streaming people" to find.
If you guys really prefer different naming then that's fine and I'll rename
everything to make it consistent with the rest of HBase.
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298975#comment-16298975
]
Jerry He commented on HBASE-19486:
--
Maybe similar to {{HRegion.MEMSTORE_PERIODIC_FLUSH_INTERVAL}}?
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298969#comment-16298969 ] Jerry He commented on HBASE-19486: -- +1 Nice! Good tests too. nit. Could you rename it to periodic flush or similar? If I recall, there is somewhere we use auto flush for a different meaning. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298904#comment-16298904 ] stack commented on HBASE-19486: --- +1 Needs release note so folks have a chance of finding this new, useful addition. Thanks [~nielsbasjes] > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298830#comment-16298830
]
Hadoop QA commented on HBASE-19486:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
28s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} hbase-client: The patch generated 0 new + 24
unchanged - 6 fixed = 24 total (was 30) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
16s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 32s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
1s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}111m 31s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903068/HBASE-19486-20171220-1612-trigger-qa-run.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8f8f6b5cb74a 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 55fefd4b5a |
| maven |
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298637#comment-16298637 ] Ted Yu commented on HBASE-19486: TestDLSAsyncFSWAL should not be related to your patch. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298564#comment-16298564
]
Niels Basjes commented on HBASE-19486:
--
The failure here was logged as
{code}
Failed to read test report file
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/hbase-server/target/surefire-reports/TEST-org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL.xml
org.dom4j.DocumentException: Error on line 75 of document : XML document
structures must start and end within the same entity. Nested exception: XML
document structures must start and end within the same entity.
at org.dom4j.io.SAXReader.read(SAXReader.java:482)
at org.dom4j.io.SAXReader.read(SAXReader.java:343)
at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:169)
{code}
As far as I can tell this has nothing to do with my patch. Correct?
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296750#comment-16296750
]
Hadoop QA commented on HBASE-19486:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
59s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} hbase-client: The patch generated 0 new + 24
unchanged - 6 fixed = 24 total (was 30) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
36s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 47s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 44s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12902824/HBASE-19486-20171219-1122-trigger-qa-run.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 98e60b4ff074 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 03e79b7994 |
| maven | version:
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296729#comment-16296729 ] Niels Basjes commented on HBASE-19486: -- TODO for consistency: - setWriteBufferMaxLinger --> setWriteBufferMaxLingerMs - writeBufferMaxLinger --> writeBufferMaxLingerMs > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296598#comment-16296598 ] Ted Yu commented on HBASE-19486: Yes > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296596#comment-16296596 ] Niels Basjes commented on HBASE-19486: -- Ok. Is something like changing a comment somewhere in hbase-server enough? > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296594#comment-16296594 ] Ted Yu commented on HBASE-19486: Mind introducing trivial change to hbase-server module to trigger QA run ? > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296588#comment-16296588
]
Hadoop QA commented on HBASE-19486:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} hbase-client: The patch generated 0 new + 24
unchanged - 6 fixed = 24 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 11s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
42s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12902808/HBASE-19486-20171219-1026.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 85b2e715491d 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 03e79b7994 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10553/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10553/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
>
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296518#comment-16296518
]
Hadoop QA commented on HBASE-19486:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
29s{color} | {color:red} hbase-client: The patch generated 2 new + 24 unchanged
- 6 fixed = 26 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
44s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 36s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12902796/HBASE-19486-20171219-0933.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6f0de242be35 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 7a7e55b601 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10552/artifact/patchprocess/diff-checkstyle-hbase-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10552/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10552/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Automatically flush
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294895#comment-16294895
]
Hadoop QA commented on HBASE-19486:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s{color}
| {color:red} HBASE-19486 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.6.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12902635/HBASE-19486-20171218-1300.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10528/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294862#comment-16294862
]
Hadoop QA commented on HBASE-19486:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HBASE-19486 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.6.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12902628/HBASE-19486-20171218-1229.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10525/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288312#comment-16288312
]
Hadoop QA commented on HBASE-19486:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
57s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
29s{color} | {color:red} hbase-client: The patch generated 8 new + 49 unchanged
- 1 fixed = 57 total (was 50) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
59m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
56s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12901758/HBASE-19486-20171212-2117.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7dd11530e557 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 11467ef111 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10391/artifact/patchprocess/diff-checkstyle-hbase-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10391/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10391/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288258#comment-16288258
]
Ted Yu commented on HBASE-19486:
{code}
+throw new UnsupportedOperationException("The
BufferedMutator::setWriteBufferMaxLinger has not been implemented");
{code}
The double colon seems to be from C++.
Please mention MIN_WRITE_BUFFER_MAX_LINGER in the javadoc.
{code}
+ default long getWriteBufferMaxLinger() {
{code}
It would be clearer if the time unit is part of the method name.
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288217#comment-16288217 ] Niels Basjes commented on HBASE-19486: -- [~yuzhih...@gmail.com] I have attached the changes as a diff/patch file. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287817#comment-16287817 ] Ted Yu commented on HBASE-19486: Niels: HBase doesn't accept pull request. Please attach patch here. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
