[
https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204385#comment-17204385
]
ASF GitHub Bot commented on PHOENIX-6153:
-----------------------------------------
stoty commented on pull request #902:
URL: https://github.com/apache/phoenix/pull/902#issuecomment-701098943
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 5m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files
found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any
anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any
@author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
||| _ 4.x Compile Tests _ |
| +1 :green_heart: | mvninstall | 10m 43s | 4.x passed |
| +1 :green_heart: | compile | 1m 11s | 4.x passed |
| +1 :green_heart: | checkstyle | 0m 38s | 4.x passed |
| +1 :green_heart: | javadoc | 1m 0s | 4.x passed |
| +0 :ok: | spotbugs | 3m 46s | phoenix-core in 4.x has 957 extant
spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 56s | the patch passed |
| +1 :green_heart: | compile | 1m 17s | the patch passed |
| +1 :green_heart: | javac | 1m 17s | the patch passed |
| -1 :x: | checkstyle | 0m 39s | phoenix-core: The patch generated 6
new + 94 unchanged - 4 fixed = 100 total (was 98) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace
issues. |
| +1 :green_heart: | javadoc | 0m 57s | the patch passed |
| +1 :green_heart: | spotbugs | 4m 0s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 153m 53s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate
ASF License warnings. |
| | | 193m 9s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.end2end.UpsertSelectIT |
| | phoenix.end2end.SystemCatalogIT |
| | phoenix.end2end.index.IndexVerificationOutputRepositoryIT |
| | phoenix.end2end.ViewMetadataIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-902/1/artifact/yetus-general-check/output/Dockerfile
|
| GITHUB PR | https://github.com/apache/phoenix/pull/902 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti
checkstyle compile |
| uname | Linux 88b7c76073ba 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | 4.x / 1df0cc9 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle |
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-902/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
|
| unit |
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-902/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
|
| Test Results |
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-902/1/testReport/
|
| Max. process+thread count | 6409 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output |
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-902/1/console
|
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Table Map Reduce job after a Snapshot based job fails with
> CorruptedSnapshotException
> -------------------------------------------------------------------------------------
>
> Key: PHOENIX-6153
> URL: https://issues.apache.org/jira/browse/PHOENIX-6153
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 4.15.0, 4.14.3, master
> Reporter: Saksham Gangwar
> Assignee: Saksham Gangwar
> Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch,
> PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch,
> PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch
>
>
> Different MR job requests which reach [MapReduceParallelScanGrouper
> getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65]
> we currently make use of shared configuration among jobs to figure out
> snapshot names.
> Example jobs' sequence: first two jobs work over snapshot and the third job
> over a regular table.
> Prininting hashcode of objects when entering:
> [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65]
> *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful)
> context.getConnection(): 521093916
> ConnectionQueryServices: 1772519705
> *Configuration conf: 813285994*
> conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1*
>
> *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful)
> context.getConnection(): 1928017473
> ConnectionQueryServices: 961279422
> *Configuration conf: 813285994*
> conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2*
>
> *Job 3:* (over the table *ABC_TABLE_3* but fails with
> CorruptedSnapshotException while it got nothing to do with snapshot)
> context.getConnection(): 28889670
> ConnectionQueryServices: 424389847
> *Configuration: 813285994*
> conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2*
>
> Exception which we get:
> [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR]
> [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read
> snapshot info
> from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo
> at
> org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.BaseResultIterators.<init>(BaseResultIterators.java:511)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.iterate.ParallelIterators.<init>(ParallelIterators.java:62)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:213)
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.setupParallelScansWithScanGrouper(PhoenixInputFormat.java:252)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.setupParallelScansFromQueryPlan(PhoenixInputFormat.java:235)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.generateSplits(PhoenixInputFormat.java:94)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:89)
>
> ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> ~[hadoop-mapreduce-client-core-2.7.7-sfdc-1.0.18.jar:2.7.7-sfdc-1.0.18]
> at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_172]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_172]
>
>
> Change Required:
> 1. While setting the snapshot name in a shared configuration we also need to
> add a mechanism to remove it as well when jobs are not snapshot related:
> [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java#L210]
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
