[jira] [Created] (HBASE-26752) Fix flappy test TestSimpleRegionNormalizerOnCluster.java
Aman Poonia created HBASE-26752: --- Summary: Fix flappy test TestSimpleRegionNormalizerOnCluster.java Key: HBASE-26752 URL: https://issues.apache.org/jira/browse/HBASE-26752 Project: HBase Issue Type: Bug Components: Normalizer Affects Versions: 1.7.1 Reporter: Aman Poonia Assignee: Aman Poonia TestSimpleRegionNormalizerOnCluster.java can hang after HBASE-26744 The assumption that order of HTable list is sorted is wrong so depending on that order can cause the test to hang or be inaccurate -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: New dedicated Jenkins Conrtoller
Most of the work is done. There is still an arm jenkins job on ci-hadoop, as we do not have arm nodes on ci-hbase. Another problem is that it seems the github web hook will not notify ci-hbase, so we need to wait for the pre-commit job to scan the github repo for triggering a build. Will keep an eye on it and work together with infra to solve it if possible. Thanks. 张铎(Duo Zhang) 于2022年2月10日周四 11:16写道： > Some updates. > > For nightly build we are still suffering from the python problem... > Anyway, using python2 directly can solve the problem. > > I'm still working on it. > > Thanks. > > 张铎(Duo Zhang) 于2022年2月9日周三 09:34写道： > >> The new Jenkins jobs are still unavailable due to lack of python support. >> >> If this still can not be resolved tonight, I will rollback to ci-hadoop >> and try migration next time. >> >> Thanks. >> >> 张铎(Duo Zhang) 于2022年2月8日周二 21:35写道： >> >>> Hi Gavin, the ssh transfer plugin is fine, but now we miss python on >>> hbase11-20. >>> >>> I've filed INFRA-22855, PTAL. >>> >>> Thanks. >>> >>> 张铎(Duo Zhang) 于2022年2月8日周二 20:19写道： >>> Thank you Gavin. WIll take a look soon! Gavin McDonald 于2022年2月8日周二 18:31写道： > Give it a try now please > > On Sat, Feb 5, 2022 at 8:55 AM 张铎(Duo Zhang) > wrote: > > > Hi, Gavin, any luck on the ssh transfer plugin? > > > > Thanks. > > > > 张铎(Duo Zhang) 于2022年1月29日周六 14:53写道： > > > >> So any updates here Gavin? > >> > >> Is the ssh transfer plugin available now? > >> > >> Thanks. > >> > >> Sean Busbey 于2022年1月23日周日 09:04写道： > >> > >>> It looks like I currently do have access to manage the ci-hbase > >>> controller. > >>> > >>> > >>> Could you check that the git-websites label handling where a common > >>> machine gets leased to the ci-hbase controller is set up properly? > or give > >>> me a pointer on how I would check this myself? > >>> > >>> I started trying to migrate our website generation job again and > it's > >>> been waiting for a node with the git-websites label to show up for > ~8 hours > >>> at this point. > >>> > >>> https://ci-hbase.apache.org/job/hbase_generate_website/3/ > >>> > >>> > >>> > >>> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald < > gmcdon...@apache.org> > >>> wrote: > >>> > Hi, > Timestamper plugin installed. > > By the way I thought I had given the PMC members permissions to > manage > Jenkins > on ci-hbase including installing plugins etc, can someone check? > > > On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) < > palomino...@gmail.com> > wrote: > > > We faced a strange issue.. > > > > rg.codehaus.groovy.control.MultipleCompilationErrorsException: > startup failed: > > WorkflowScript: 29: Invalid option type "timestamps". Valid > option > types: [buildDiscarder, catchError, checkoutToSubdirectory, > disableConcurrentBuilds, disableResume, durabilityHint, > githubProjectProperty, newContainerPerStage, > overrideIndexTriggers, > parallelsAlwaysFailFast, preserveStashes, quietPeriod, > rateLimitBuilds, > retry, script, skipDefaultCheckout, skipStagesAfterUnstable, > timeout, > waitUntil, warnError, withChecks, withContext, withCredentials, > withEnv, > wrap, ws] @ line 29, column 5. > >timestamps() > >^ > > > > 1 error > > > > at > > org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) > > at > > org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) > > at > > org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) > > at > > org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) > > at > > org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) > > at > > groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) > > at > > groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) > > at > groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) > > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) > > at > > org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) > > at > > org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) > > at > >
[jira] [Resolved] (HBASE-26749) Migrate HBase main pre commit job to ci-hbase
[ https://issues.apache.org/jira/browse/HBASE-26749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26749. --- Fix Version/s: 2.5.0 1.7.2 3.0.0-alpha-3 2.4.10 Hadoop Flags: Reviewed Resolution: Fixed Merged to all ative branches. Thanks [~busbey] for reviewing! > Migrate HBase main pre commit job to ci-hbase > - > > Key: HBASE-26749 > URL: https://issues.apache.org/jira/browse/HBASE-26749 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 1.7.2, 3.0.0-alpha-3, 2.4.10 > > > We used to run this on the hadoop label so first we need to change the label > in jenkins file. > And before migration, we need to move the 10 hbase nodes from ci-hadoop to > ci-hbase as there are no jobs on them any more. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26697) Migrate HBase Nightly HBase-Flaky-Tests and HBase-Find-Flaky-Tests to ci-hbase
[ https://issues.apache.org/jira/browse/HBASE-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26697. --- Fix Version/s: 2.5.0 1.7.2 3.0.0-alpha-3 2.4.10 Hadoop Flags: Reviewed Resolution: Fixed The work is done. Thanks all for helping! > Migrate HBase Nightly HBase-Flaky-Tests and HBase-Find-Flaky-Tests to ci-hbase > -- > > Key: HBASE-26697 > URL: https://issues.apache.org/jira/browse/HBASE-26697 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 1.7.2, 3.0.0-alpha-3, 2.4.10 > > > They are tied together so let's do them at once. > We also need to modify the url for our pre commit job. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26644) Spurious compaction failures with file tracker
[
https://issues.apache.org/jira/browse/HBASE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved HBASE-26644.
Resolution: Not A Problem
Yep, all good. I believe you fixed this in HBASE-26675
> Spurious compaction failures with file tracker
> --
>
> Key: HBASE-26644
> URL: https://issues.apache.org/jira/browse/HBASE-26644
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
>
> Noticed when running a basic {{{}hbase pe randomWrite{}}}, we'll see
> compactions failing at various points.
> One example:
> {noformat}
> 2022-01-03 17:41:18,319 ERROR
> [regionserver/localhost:16020-shortCompactions-0]
> regionserver.CompactSplit(670): Compaction failed
> region=TestTable,0004054490,1641249249856.2dc7251c6eceb660b9c7bb0b587db913.,
> storeName=2dc7251c6eceb660b9c7bb0b587db913/info0, priority=6,
> startTime=1641249666161
> java.io.IOException: Root-level entries already added in single-level mode
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeSingleLevelIndex(HFileBlockIndex.java:1136)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterWriter$MetaWriter.write(CompoundBloomFilterWriter.java:279)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl$1.writeToBlock(HFileWriterImpl.java:713)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeBlock(HFileBlock.java:1205)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:660)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:377)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.commitWriter(DefaultCompactor.java:70)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:386)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:62)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:125)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1141)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2388)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:654)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:697)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {noformat}
> This isn't a super-critical issue because compactions will be retried
> automatically and they appear to eventually succeed. However, when the max
> storefiles limit is reaching, this does cause ingest to hang (as I was doing
> with my modest configuration).
> We had seen a similar kind of problem in our testing when backporting to
> HBase 2.4 (not upstream as the decision was to not do this) which we
> eventually tracked down to a bad merge-conflict resolution to the new HFile
> Cleaner. However, initial investigations don't have the same exact problem.
> It seems that we have some kind of generic race condition. Would be good to
> add more logging to catch this in the future (since we have two separate
> instances of this category of bug already).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26750) The branch-1 nightly jenkins job fails with "No such DSL method 'jiraComment' found"
[ https://issues.apache.org/jira/browse/HBASE-26750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26750. --- Assignee: Duo Zhang Resolution: Fixed I installed jira plugin on ci-hbase and also configured the jira site. Now it works, see HBASE-26747. Resolved. > The branch-1 nightly jenkins job fails with "No such DSL method 'jiraComment' > found" > > > Key: HBASE-26750 > URL: https://issues.apache.org/jira/browse/HBASE-26750 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-24146) Run a perf comparison of 2.3.x with 2.2.x
[ https://issues.apache.org/jira/browse/HBASE-24146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24146. -- Resolution: Won't Do > Run a perf comparison of 2.3.x with 2.2.x > - > > Key: HBASE-24146 > URL: https://issues.apache.org/jira/browse/HBASE-24146 > Project: HBase > Issue Type: Task > Components: community, Performance >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Priority: Major > > Kick the tires on a recent build. Compare results with a run on the latest > 2.2.x. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26741) Incorrect exception handling in shell
[
https://issues.apache.org/jira/browse/HBASE-26741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-26741.
---
Fix Version/s: 2.5.0
3.0.0-alpha-3
2.4.10
Resolution: Fixed
Merged to branch-2.4+. Thanks for the review [~elserj].
Filed HBASE-26751 to cover the shell exit behavior with tests.
> Incorrect exception handling in shell
> -
>
> Key: HBASE-26741
> URL: https://issues.apache.org/jira/browse/HBASE-26741
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.4.10
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
> The exception handling changed in the shell compared to 2.2.
> {noformat}
> ➜ hbase-upstream git:(branch-2.4) ✗ cat commands.txt
> scan 'foo'
> exit
> ➜ hbase-upstream git:(branch-2.4) ✗ bin/hbase shell -n commands.txt
> 2022-02-07 16:21:33,654 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hbase:001:0> scan 'foo'
> ROW COLUMN+CELL
> Took 0.3890 seconds
> Traceback (most
> recent call last):
> RuntimeError (Unknown table foo!)
> hbase:002:0> exit
> ➜ hbase-upstream git:(branch-2.4) ✗ echo $?
> 0 {noformat}
> The execution continues even after an exception is thrown. In 2.2.7 the
> execution stops when an exception is thrown and the shell exits with error
> code.
> {noformat}
> ➜ hbase-2.2.7 bin/hbase shell -n commands.txt
> 2022-02-07 16:33:54,930 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ROW COLUMN+CELL
> Took 0.3082 seconds
> RuntimeError:
> Unknown table foo!
> translate_hbase_exceptions at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:130
> command_safe at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:49
> internal_command at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:148
> command at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:140
> scan at (eval):2
> at commands.txt:1
> load at org/jruby/RubyKernel.java:973
> at
> /Users/petersomogyi/tmp/hbase-2.2.7/bin/../bin/hirb.rb:186
> ➜ hbase-2.2.7 echo $?
> 1 {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26751) Tests for shell exit behavior
Peter Somogyi created HBASE-26751: - Summary: Tests for shell exit behavior Key: HBASE-26751 URL: https://issues.apache.org/jira/browse/HBASE-26751 Project: HBase Issue Type: Test Components: shell, test Reporter: Peter Somogyi HBase shell exit behavior and exception handling is not covered with tests. The tests should cover the clean exits and also when a command throws exceptions in interactive and non-interactive modes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26744) Normalizer exits without normalizing all the tables
[
https://issues.apache.org/jira/browse/HBASE-26744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha resolved HBASE-26744.
Fix Version/s: 1.8.0
Resolution: Fixed
Merged to branch-1, thanks [~mnpoonia] for contributing.
> Normalizer exits without normalizing all the tables
> ---
>
> Key: HBASE-26744
> URL: https://issues.apache.org/jira/browse/HBASE-26744
> Project: HBase
> Issue Type: Bug
> Components: Normalizer
>Affects Versions: 1.7.1
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Minor
> Fix For: 1.8.0
>
>
> Currently if there are multiple tables to normalize, normalizer is exiting
> even before iterating all the table if there is a table that doesnot require
> normalization
> here is the offending code
> [https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1736]
>
> {code:java}
> List plans = this.normalizer.computePlansForTable(table);
>
> if (plans == null || plans.isEmpty()) {
>return true;
> } {code}
> this is running inside loop over tables
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
