[
https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201812#comment-17201812
]
Hadoop QA commented on PHOENIX-6153:
------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
46s{color} | {color:blue} phoenix-core in master has 973 extant spotbugs
warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
31s{color} | {color:red} phoenix-core: The patch generated 7 new + 93 unchanged
- 4 fixed = 100 total (was 97) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 10s{color}
| {color:red} phoenix-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/PreCommit-PHOENIX-Build/45/artifact/patchprocess/Dockerfile
|
| JIRA Issue | PHOENIX-6153 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13012095/PHOENIX-6153.master.v4.patch
|
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti
checkstyle compile |
| uname | Linux 8c760050b3a2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / d14e3d0 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle |
https://ci-hadoop.apache.org/job/PreCommit-PHOENIX-Build/45/artifact/patchprocess/diff-checkstyle-phoenix-core.txt
|
| unit |
https://ci-hadoop.apache.org/job/PreCommit-PHOENIX-Build/45/artifact/patchprocess/patch-unit-phoenix-core.txt
|
| Test Results |
https://ci-hadoop.apache.org/job/PreCommit-PHOENIX-Build/45/testReport/ |
| Max. process+thread count | 6817 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-PHOENIX-Build/45/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Table Map Reduce job after a Snapshot based job fails with
> CorruptedSnapshotException
> -------------------------------------------------------------------------------------
>
> Key: PHOENIX-6153
> URL: https://issues.apache.org/jira/browse/PHOENIX-6153
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 4.14.3, 4.x, master
> Reporter: Saksham Gangwar
> Assignee: Saksham Gangwar
> Priority: Major
> Fix For: 4.16.0
>
> Attachments: PHOENIX-6153.master.v1.patch,
> PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch,
> PHOENIX-6153.master.v4.patch
>
>
> Different MR job requests which reach [MapReduceParallelScanGrouper
> getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65]
> we currently make use of shared configuration among jobs to figure out
> snapshot names.
> Example jobs' sequence: first two jobs work over snapshot and the third job
> over a regular table.
> Prininting hashcode of objects when entering:
> [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65]
> *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful)
> context.getConnection(): 521093916
> ConnectionQueryServices: 1772519705
> *Configuration conf: 813285994*
> conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1*
>
> *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful)
> context.getConnection(): 1928017473
> ConnectionQueryServices: 961279422
> *Configuration conf: 813285994*
> conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2*
>
> *Job 3:* (over the table *ABC_TABLE_3* but fails with
> CorruptedSnapshotException while it got nothing to do with snapshot)
> context.getConnection(): 28889670
> ConnectionQueryServices: 424389847
> *Configuration: 813285994*
> conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2*
>
> Exception which we get:
> [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR]
> [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read
> snapshot info
> from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo
> at
> org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.<init>(BaseResultIterators.java:511)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.ParallelIterators.<init>(ParallelIterators.java:62)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:213)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.setupParallelScansWithScanGrouper(PhoenixInputFormat.java:252)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.setupParallelScansFromQueryPlan(PhoenixInputFormat.java:235)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.generateSplits(PhoenixInputFormat.java:94)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:89)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_172]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_172]
>
>
> Change Required:
> 1. While setting the snapshot name in a shared configuration we also need to
> add a mechanism to remove it as well when jobs are not snapshot related:
> [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java#L210]
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
