[PR] Error from set future [accumulo]
dlmarion opened a new pull request, #4442: URL: https://github.com/apache/accumulo/pull/4442 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] Possible tablet refresh causes scan on metadata table to fail [accumulo]
dlmarion opened a new issue, #4441: URL: https://github.com/apache/accumulo/issues/4441 Stack trace below found in log during SplitMillion IT. The client in this case was scanning the metadata table and the tablet server hosting the metadata table logged this error. Around the same time the metadata table had compacted and a tablet metadata refresh operation occurred, which could be the cause of this. ``` java.lang.IllegalStateException: Called next() when there is no top at org.apache.accumulo.core.iteratorsImpl.system.HeapIterator.next(HeapIterator.java:72) ~[classes/:?] at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:101) ~[classes/:?] at org.apache.accumulo.tserver.MemKeyConversionIterator.next(MemKeyConversionIterator.java:73) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:179) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:149) ~[classes/:?] at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:101) ~[classes/:?] at org.apache.accumulo.core.iterators.SkippingIterator.next(SkippingIterator.java:37) ~[classes/:?] at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:101) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.HeapIterator.next(HeapIterator.java:75) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.StatsIterator.next(StatsIterator.java:51) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.DeletingIterator.next(DeletingIterator.java:65) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.ColumnFamilySkippingIterator.consume(ColumnFamilySkippingIterator.java:66) ~[classes/:?] at org.apache.accumulo.core.iterators.ServerSkippingIterator.next(ServerSkippingIterator.java:46) ~[classes/:?] at org.apache.accumulo.core.iterators.ServerFilter.next(ServerFilter.java:51) ~[classes/:?] at org.apache.accumulo.core.iterators.SynchronizedServerFilter.next(SynchronizedServerFilter.java:51) ~[classes/:?] at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:101) ~[classes/:?] at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103) ~[classes/:?] at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:60) ~[classes/:?] at org.apache.accumulo.core.iterators.user.RowFilter.skipRows(RowFilter.java:118) ~[classes/:?] at org.apache.accumulo.core.iterators.user.RowFilter.seek(RowFilter.java:191) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:165) ~[classes/:?] at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.seek(SourceSwitchingIterator.java:237) ~[classes/:?] at org.apache.accumulo.tserver.tablet.TabletBase.nextBatch(TabletBase.java:280) ~[classes/:?] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Add blog post for Compactor process memory testing [accumulo-website]
DomGarguilo opened a new pull request, #421: URL: https://github.com/apache/accumulo-website/pull/421 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] verifies tablets are seen by compaction driver [accumulo]
keith-turner merged PR #4434: URL: https://github.com/apache/accumulo/pull/4434 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] conditionally removes tables scan files [accumulo]
keith-turner merged PR #4436: URL: https://github.com/apache/accumulo/pull/4436 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (ACCUMULO-1703) Thrift generated code isn't properly checked for optional fields
[ https://issues.apache.org/jira/browse/ACCUMULO-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834988#comment-17834988 ] Christopher Tubbs commented on ACCUMULO-1703: - This can be fixed by the mandatory use of Thrift private members, as per https://github.com/apache/accumulo/pull/3405 > Thrift generated code isn't properly checked for optional fields > > > Key: ACCUMULO-1703 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1703 > Project: Accumulo > Issue Type: Bug >Reporter: Christopher Tubbs >Priority: Minor > > In our thrift definition files, we declare some fields as optional. However, > we don't actually check to see if they are sent. This may be okay for > Objects, which we can check for null, but it won't work for checking optional > primitives. > Example: > ProxyServer.createBatchScanner() accepts a BatchScanOptions object, which has > the number of threads as an optional parameter. Instead of calling > opts.threads, we should check if opts.isSetThreads() is true, and then > retrieve it with opts.getThreads(). > This is an essential check for all optional arguments (non-optional args are > validated by the generated validate() method). > The importance of the optional field is high, if we want to support RPC > compatibility across versions. As far as I know, this is one of the goals of > the client proxy. So, we must use the thrift features (such as the optional > flag) correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-1703) Thrift generated code isn't properly checked for optional fields
[ https://issues.apache.org/jira/browse/ACCUMULO-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1703. - Resolution: Duplicate > Thrift generated code isn't properly checked for optional fields > > > Key: ACCUMULO-1703 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1703 > Project: Accumulo > Issue Type: Bug >Reporter: Christopher Tubbs >Priority: Minor > > In our thrift definition files, we declare some fields as optional. However, > we don't actually check to see if they are sent. This may be okay for > Objects, which we can check for null, but it won't work for checking optional > primitives. > Example: > ProxyServer.createBatchScanner() accepts a BatchScanOptions object, which has > the number of threads as an optional parameter. Instead of calling > opts.threads, we should check if opts.isSetThreads() is true, and then > retrieve it with opts.getThreads(). > This is an essential check for all optional arguments (non-optional args are > validated by the generated validate() method). > The importance of the optional field is high, if we want to support RPC > compatibility across versions. As far as I know, this is one of the goals of > the client proxy. So, we must use the thrift features (such as the optional > flag) correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-3121) Add ability to watch configuration changes to API
[ https://issues.apache.org/jira/browse/ACCUMULO-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3121. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Add ability to watch configuration changes to API > - > > Key: ACCUMULO-3121 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3121 > Project: Accumulo > Issue Type: Improvement > Components: client >Reporter: Christopher Tubbs >Priority: Minor > > In ACCUMULO-2841, an improvement was proposed by [~vines] to have the ability > to watch certain configuration attributes. > We currently don't support this in our API, but in theory, it's possible, > because we store these things in ZooKeeper, which does support watchers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] batches reading tablet metadata for refresh [accumulo]
keith-turner commented on code in PR #4439:
URL: https://github.com/apache/accumulo/pull/4439#discussion_r1556093180
##
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java:
##
@@ -132,6 +132,7 @@ public class Tablet extends TabletBase {
private final AtomicLong dataSourceDeletions = new AtomicLong(0);
private volatile TabletMetadata latestMetadata;
+ private long refreshCount = 0;
Review Comment:
I had not made that volatile because it was being read an written while a
lock was held, so it did not need to be volatile. The reason I was reading it
with the lock held is because it was dependent on another variable and having
two volatiles that depend on each other is really tricky and something I like
to avoid. Realized I could put both variables under a single volatile to
remove this tricky part and allow reading safely outside of a lock. Made that
change in bb8ae31.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (ACCUMULO-671) Create utility to re-initialize just ZooKeeper from existing Accumulo install
[ https://issues.apache.org/jira/browse/ACCUMULO-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-671. Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Create utility to re-initialize just ZooKeeper from existing Accumulo install > - > > Key: ACCUMULO-671 > URL: https://issues.apache.org/jira/browse/ACCUMULO-671 > Project: Accumulo > Issue Type: New Feature > Components: master >Reporter: Krishmin Rai >Priority: Minor > > In a case where ZooKeeper data is lost but the Accumulo footprint in HDFS > still exists, it would be nice to be able to properly re-initialize just > ZooKeeper (to the extent possible). > See [mailing > list|http://mail-archives.apache.org/mod_mbox/accumulo-user/201207.mbox/%3CCAGUtCHrfNCApFb5iAVJX9R%3DuzwP5zT7FHw7zBrytMQwH1voJcw%40mail.gmail.com%3E] > for discussion -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-463) use term cardinalities to optimize the AndIterator
[ https://issues.apache.org/jira/browse/ACCUMULO-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-463. Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > use term cardinalities to optimize the AndIterator > -- > > Key: ACCUMULO-463 > URL: https://issues.apache.org/jira/browse/ACCUMULO-463 > Project: Accumulo > Issue Type: Improvement > Components: wikisearch >Reporter: Eric C. Newton >Priority: Minor > > The wikisearch example stores term cardinalities as metadata, but does not > use the term frequency to determine the best initial seek term for > conjunctions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-1344) Shell gives erroneous error messages during long compaction
[
https://issues.apache.org/jira/browse/ACCUMULO-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-1344.
-
Resolution: Abandoned
Closing old inactive issue; if this is still a problem, please create a new
issue or pull request at https://github.com/apache/accumulo
> Shell gives erroneous error messages during long compaction
> ---
>
> Key: ACCUMULO-1344
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1344
> Project: Accumulo
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.4.2
>Reporter: Mike Drob
>Priority: Minor
>
> When waiting for a large compaction to finish that was kicked off in the
> shell via {{compact -w}}, it may warn that {{Thread "shell" stuck on IO}}
> after ten minutes.
> This warning may be misleading, as the compaction could be running just fine,
> and simply running over a large table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-3088) StatsIterator readCounter could be more accurate
[ https://issues.apache.org/jira/browse/ACCUMULO-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3088. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > StatsIterator readCounter could be more accurate > > > Key: ACCUMULO-3088 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3088 > Project: Accumulo > Issue Type: Bug >Reporter: Adam Fuchs >Priority: Trivial > Labels: newbie > > When testing seek performance I noticed that the read count displayed on > Accumulo's monitor page registered zero entries read per second and a > positive number of entries per second returned. This should not be possible. > There is a fencepost error in the StatsIterator where seeks don't count as > entries read. > Successful completion of this ticket should handle the cases where a seek > returns no entries, as well as where next is never called on an iterator. > The cost of incrementing an AtomicLong is something like 10ns, which is at > least an order of magnitude more than the desired overhead. The current way > of batching updates to the read counter should be preserved for performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-2953) Maxima in memory stress test are one less than configuration options
[
https://issues.apache.org/jira/browse/ACCUMULO-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2953.
-
Resolution: Abandoned
Closing old inactive issue; if this is still a problem, please create a new
issue or pull request at https://github.com/apache/accumulo
> Maxima in memory stress test are one less than configuration options
>
>
> Key: ACCUMULO-2953
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2953
> Project: Accumulo
> Issue Type: Test
> Components: test
>Reporter: Bill Havanki
>Priority: Trivial
> Labels: boundary, test
>
> The {{RandomWithinRange}} class in the memory stress test takes care of
> generating random integers across a range. The range is inclusive at the
> beginning and exclusive at the end. That means that the maxima for various
> test configuration options, such as key component and value lengths, are
> actually one less than the option values. For example, with
> --min-value-size=1 and --max-value-size=10, value lengths will range only
> from 1 to 9.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-2055) configure random walkers to send logs to the monitor
[ https://issues.apache.org/jira/browse/ACCUMULO-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2055. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo (Also, this should be easily doable with the latest monitor appender configured by default in our reference log4j config for servers) > configure random walkers to send logs to the monitor > > > Key: ACCUMULO-2055 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2055 > Project: Accumulo > Issue Type: Improvement > Components: test >Reporter: Eric C. Newton >Priority: Minor > Labels: newbie > > There's a bit of a hack to configure WARN/ERROR messages to a shared NFS file > system so that you can monitor all the walkers from one place. However, if > you don't have a shared NFS, it's painful to monitor the logs distributed > throughout the cluster. We already have a single place to watch the logs: > the monitor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-4559) Create status API for Accumulo services
[ https://issues.apache.org/jira/browse/ACCUMULO-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4559. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Create status API for Accumulo services > --- > > Key: ACCUMULO-4559 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4559 > Project: Accumulo > Issue Type: Improvement > Components: gc, master, tserver >Reporter: Luis Tavarez >Priority: Minor > > Create a status API to the accumulo services, similar to the tservers status. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-4493) Shell should be able to use keytab login
[ https://issues.apache.org/jira/browse/ACCUMULO-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4493. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Shell should be able to use keytab login > > > Key: ACCUMULO-4493 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4493 > Project: Accumulo > Issue Type: New Feature > Components: shell >Reporter: Sean Busbey >Priority: Minor > > Users should be able to launch the shell in a kerberos deployment using a > keytab. > current workaround: use the system shell to kinit with the keytab, then > launch the shell, then kdestroy > Workaround doesn't allow re-login from keytab for long running shell. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-4595) Simple validator for inserting data
[ https://issues.apache.org/jira/browse/ACCUMULO-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4595. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Simple validator for inserting data > --- > > Key: ACCUMULO-4595 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4595 > Project: Accumulo > Issue Type: New Feature >Reporter: Michael Wall >Priority: Minor > > It would be useful to be able to write custom validation for specific parts > of a Key or Value that could be applied to both mutations on live ingest and > writing RFiles for bulk loads. Initially I thought this could be done with a > table constraint, but didn't consider the importdirectory use case where > RFiles are bulk imported. This validation does not need to track state > though. Thinking this is really only for simple validation. Couple of > contrived examples. > 1) Validate the CF starts with the string "abcd". > 2) Validate the CV only contains strings from a predefined list of strings. > 3) Validate the row only contains letters or numbers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-2596) Document failure mode around persistent version upgrades
[ https://issues.apache.org/jira/browse/ACCUMULO-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2596. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Document failure mode around persistent version upgrades > > > Key: ACCUMULO-2596 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2596 > Project: Accumulo > Issue Type: Task > Components: docs, master >Affects Versions: 1.4.5, 1.5.1, 1.6.0 >Reporter: Sean Busbey >Priority: Minor > Labels: upgrade > > updateAccumuloVersion does a non-atomic update of the persisted version, and > getPersistentVersion relies on the ordering of the underlying fs/Volume to > determine which version to return. > We should document what happens when there's a partial failure, improve > reporting in it to the operator, and document how to fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-2607) Add "list tables" flag to LogReader
[ https://issues.apache.org/jira/browse/ACCUMULO-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2607. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Add "list tables" flag to LogReader > --- > > Key: ACCUMULO-2607 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2607 > Project: Accumulo > Issue Type: Improvement > Components: logger >Affects Versions: 1.4.5, 1.5.1, 1.6.0 >Reporter: Sean Busbey >Priority: Minor > Labels: newbie > > Add a flag to LogReader that has it just print out the list of tables that > have entries in a passed WAL. > I was doing an upgrade test for ACCUMULO-2519, and making sure I had WALs > needed to recover the !METADATA table would have been much easier with this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-2772) Bulk Import of incorrect RFile version should give error message that explains incompatibility
[
https://issues.apache.org/jira/browse/ACCUMULO-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2772.
-
Resolution: Abandoned
Closing old inactive issue; if this is still a problem, please create a new
issue or pull request at https://github.com/apache/accumulo
> Bulk Import of incorrect RFile version should give error message that
> explains incompatibility
> --
>
> Key: ACCUMULO-2772
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2772
> Project: Accumulo
> Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0, 1.6.0
>Reporter: Sean Busbey
>Priority: Minor
> Labels: error-feedback, logging
>
> In ACCUMULO-1079 this comment showed incorrect error handling
> {quote}
> David Medinets added a comment - 19 minutes ago
> ~[busbey] the error did not mention rfile version mismatch. There was no
> error at the command line. The rfile was moved to the failures directory and
> the 'unable to find tablets that overlap file' message was in the logs.
> {quote}
> We should make sure that we correctly report RFile mismatches during bulk
> load attempts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-4560) Create status API for Master
[ https://issues.apache.org/jira/browse/ACCUMULO-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4560. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Create status API for Master > > > Key: ACCUMULO-4560 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4560 > Project: Accumulo > Issue Type: Sub-task > Components: master >Reporter: Luis Tavarez >Priority: Minor > > Create status API for Master, to include Accumulo version of master. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-1512) Provide example using KeyRangePartitioner
[ https://issues.apache.org/jira/browse/ACCUMULO-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1512. - Resolution: Abandoned Closing old inactive issue; if this is still a problem, please create a new issue or pull request at https://github.com/apache/accumulo > Provide example using KeyRangePartitioner > - > > Key: ACCUMULO-1512 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1512 > Project: Accumulo > Issue Type: Improvement >Affects Versions: 1.4.1, 1.5.0, 1.6.0 >Reporter: David Medinets >Priority: Minor > Labels: Documentation > > The KeyRangePartitioner class lets mappers output Accumulo Key objects > instead of Text. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ACCUMULO-1543) tablets without WALs can be loaded in parallel
[ https://issues.apache.org/jira/browse/ACCUMULO-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1543. - Resolution: Duplicate Closing as duplicate of https://github.com/apache/accumulo/pull/4429 > tablets without WALs can be loaded in parallel > -- > > Key: ACCUMULO-1543 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1543 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Reporter: Eric C. Newton >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Clean tablets can be loaded in parallel. There are concerns that loading > tablets that require replay of mutations from WALs would require too much > memory (ACCUMULO-1085). However, tablets that do not refer to WALs can be > loaded quickly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] updates bulk import and compaction logging [accumulo]
keith-turner commented on code in PR #4440:
URL: https://github.com/apache/accumulo/pull/4440#discussion_r1556033757
##
server/manager/src/main/java/org/apache/accumulo/manager/tableOps/bulkVer2/LoadFiles.java:
##
@@ -283,23 +293,25 @@ long finish() {
results.values().stream().allMatch(result -> result.getStatus() ==
Status.ACCEPTED)
&& skipped == 0;
+ results.forEach((extent, condResult) -> {
+if (condResult.getStatus() == Status.ACCEPTED) {
+ loadingFiles.get(extent).forEach(file ->
TabletLogger.bulkImported(extent, file));
+} else {
+ var metadata = condResult.readMetadata();
+ if (metadata == null) {
+log.debug("Tablet update failed, tablet is gone {} {} {}", fateId,
extent,
+condResult.getStatus());
+ } else {
+log.debug("Tablet update failed {} {} {} {} {} {}", fateId, extent,
+condResult.getStatus(), metadata.getOperationId(),
metadata.getLocation(),
+metadata.getLoaded());
+ }
+}
+ });
+
Review Comment:
That is a nice improvement, updated in 48b6c83
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Stop tracking last compactor check-in for non-existent groups [accumulo]
dlmarion commented on PR #4403: URL: https://github.com/apache/accumulo/pull/4403#issuecomment-2043033672 I believe this is complete, just waiting on some changes from @ddanielr that have to do with the compaction configuration changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Modified removeUnusedWALEntries to use conditional mutations [accumulo]
dlmarion merged PR #4404: URL: https://github.com/apache/accumulo/pull/4404 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] updates bulk import and compaction logging [accumulo]
dlmarion commented on code in PR #4440:
URL: https://github.com/apache/accumulo/pull/4440#discussion_r1555795715
##
server/manager/src/main/java/org/apache/accumulo/manager/tableOps/bulkVer2/LoadFiles.java:
##
@@ -283,23 +293,25 @@ long finish() {
results.values().stream().allMatch(result -> result.getStatus() ==
Status.ACCEPTED)
&& skipped == 0;
+ results.forEach((extent, condResult) -> {
+if (condResult.getStatus() == Status.ACCEPTED) {
+ loadingFiles.get(extent).forEach(file ->
TabletLogger.bulkImported(extent, file));
+} else {
+ var metadata = condResult.readMetadata();
+ if (metadata == null) {
+log.debug("Tablet update failed, tablet is gone {} {} {}", fateId,
extent,
+condResult.getStatus());
+ } else {
+log.debug("Tablet update failed {} {} {} {} {} {}", fateId, extent,
+condResult.getStatus(), metadata.getOperationId(),
metadata.getLocation(),
+metadata.getLoaded());
+ }
+}
+ });
+
Review Comment:
I think you could probably get rid of the `allDone` variable and associated
stream operation by just setting an AtomicBoolean to `true` in the `else`
clause of the conditional below. Then after checking all results, you could
return 1000 if skipped != 0 or this new boolean is true.
Something like:
```suggestion
AtomicBoolean seenFailure = new AtomicBoolean(false);
results.forEach((extent, condResult) -> {
if (condResult.getStatus() == Status.ACCEPTED) {
loadingFiles.get(extent).forEach(file ->
TabletLogger.bulkImported(extent, file));
} else {
seenFailure.set(true);
var metadata = condResult.readMetadata();
if (metadata == null) {
log.debug("Tablet update failed, tablet is gone {} {} {}",
fateId, extent,
condResult.getStatus());
} else {
log.debug("Tablet update failed {} {} {} {} {} {}", fateId,
extent,
condResult.getStatus(), metadata.getOperationId(),
metadata.getLocation(),
metadata.getLoaded());
}
}
});
if (seenFailure.get() || skipped != 0) {
return 1000;
} else {
return 0;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] batches reading tablet metadata for refresh [accumulo]
dlmarion commented on code in PR #4439:
URL: https://github.com/apache/accumulo/pull/4439#discussion_r1555734324
##
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java:
##
@@ -132,6 +132,7 @@ public class Tablet extends TabletBase {
private final AtomicLong dataSourceDeletions = new AtomicLong(0);
private volatile TabletMetadata latestMetadata;
+ private long refreshCount = 0;
Review Comment:
should this be volatile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] catches exceptions and logs when planning compactions [accumulo]
dlmarion commented on code in PR #4435:
URL: https://github.com/apache/accumulo/pull/4435#discussion_r1555720798
##
server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionJobGenerator.java:
##
@@ -87,42 +88,61 @@ public CompactionJobGenerator(PluginEnvironment env,
unknownCompactionServiceErrorCache =
Caches.getInstance().createNewBuilder(CacheName.COMPACTION_SERVICE_UNKNOWN,
false)
.expireAfterWrite(5, TimeUnit.MINUTES).build();
+
+generateJobsErrorCache =
+
Caches.getInstance().createNewBuilder(CacheName.COMPACTION_SERVICE_UNKNOWN,
false)
+.expireAfterWrite(5, TimeUnit.MINUTES).build();
+
}
public Collection generateJobs(TabletMetadata tablet,
Set kinds) {
+try {
+ Collection systemJobs = Set.of();
-// ELASTICITY_TODO do not want user configured plugins to cause exceptions
that prevents tablets
-// from being
-// assigned. So probably want to catch exceptions and log, but not too
spammily OR some how
-// report something
-// back to the manager so it can log.
-
-Collection systemJobs = Set.of();
+ if (kinds.contains(CompactionKind.SYSTEM)) {
+CompactionServiceId serviceId = dispatch(CompactionKind.SYSTEM,
tablet, Map.of());
+systemJobs = planCompactions(serviceId, CompactionKind.SYSTEM, tablet,
Map.of());
+ }
-if (kinds.contains(CompactionKind.SYSTEM)) {
- CompactionServiceId serviceId = dispatch(CompactionKind.SYSTEM, tablet,
Map.of());
- systemJobs = planCompactions(serviceId, CompactionKind.SYSTEM, tablet,
Map.of());
-}
+ Collection userJobs = Set.of();
-Collection userJobs = Set.of();
+ if (kinds.contains(CompactionKind.USER) && tablet.getSelectedFiles() !=
null) {
+var hints =
allExecutionHints.get(tablet.getSelectedFiles().getFateId());
+if (hints != null) {
+ CompactionServiceId serviceId = dispatch(CompactionKind.USER,
tablet, hints);
+ userJobs = planCompactions(serviceId, CompactionKind.USER, tablet,
hints);
+}
+ }
-if (kinds.contains(CompactionKind.USER) && tablet.getSelectedFiles() !=
null) {
- var hints = allExecutionHints.get(tablet.getSelectedFiles().getFateId());
- if (hints != null) {
-CompactionServiceId serviceId = dispatch(CompactionKind.USER, tablet,
hints);
-userJobs = planCompactions(serviceId, CompactionKind.USER, tablet,
hints);
+ if (userJobs.isEmpty()) {
+return systemJobs;
+ } else if (systemJobs.isEmpty()) {
+return userJobs;
+ } else {
+var all = new ArrayList(systemJobs.size() +
userJobs.size());
+all.addAll(systemJobs);
+all.addAll(userJobs);
+return all;
+ }
+} catch (Exception e) {
+ // The same code that plans compactions also assigns tablets. The intent
of this catch is
+ // mainly to defend against user plugins called here that throw an
exception from negatively
+ // impacting tablet assignment.
+ String cacheKey = tablet.getTableId() + " " + e.getClass().getName();
+ // This check defends against every tablet in a table having the same
problem and therefore
+ // generating an enormous amount of spam for the logs.
+ var last = generateJobsErrorCache.getIfPresent(cacheKey);
Review Comment:
I think you might be able to just call `Cache.get(Key,Function)` here and
`last` will be `null` if it doesn't exist.
```suggestion
if (generateJobsErrorCache.get(cacheKey, () ->
System.currentTimeMillis) == null) {
```
##
server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionJobGenerator.java:
##
@@ -87,42 +88,61 @@ public CompactionJobGenerator(PluginEnvironment env,
unknownCompactionServiceErrorCache =
Caches.getInstance().createNewBuilder(CacheName.COMPACTION_SERVICE_UNKNOWN,
false)
.expireAfterWrite(5, TimeUnit.MINUTES).build();
+
+generateJobsErrorCache =
+
Caches.getInstance().createNewBuilder(CacheName.COMPACTION_SERVICE_UNKNOWN,
false)
+.expireAfterWrite(5, TimeUnit.MINUTES).build();
+
}
public Collection generateJobs(TabletMetadata tablet,
Set kinds) {
+try {
+ Collection systemJobs = Set.of();
-// ELASTICITY_TODO do not want user configured plugins to cause exceptions
that prevents tablets
-// from being
-// assigned. So probably want to catch exceptions and log, but not too
spammily OR some how
-// report something
-// back to the manager so it can log.
-
-Collection systemJobs = Set.of();
+ if (kinds.contains(CompactionKind.SYSTEM)) {
+CompactionServiceId serviceId = dispatch(CompactionKind.SYSTEM,
tablet, Map.of());
+systemJobs = planCompactions(serviceId, CompactionKind.SYSTEM, tablet,
Map.of());
+ }
-if (kinds.contains(CompactionKind.SYSTEM)) {
- CompactionServiceId
