[jira] [Commented] (SOLR-13587) Close BackupRepository after every usage
[
https://issues.apache.org/jira/browse/SOLR-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877708#comment-16877708
]
Mikhail Khludnev commented on SOLR-13587:
-
I suppose we need to go with it, enforce closing backup repoes. Need to think
how to enforce it across tests.
Here's why: HdfsBackupRepo should FS.newInstance and then it should supply it
for HDFSDirectories, and close afterward. Otherwise, it either will open (HD)FS
on the every file (that's expected to be slow) or accidentally hit FS already
closed.
Concerns, opinions?
> Close BackupRepository after every usage
>
>
> Key: SOLR-13587
> URL: https://issues.apache.org/jira/browse/SOLR-13587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Affects Versions: 8.1
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-13587.patch
>
>
> Turns out BackupRepository is created every operation, but never closed. I
> suppose it leads to necessity to have {{BadHdfsThreadsFilter}} in
> {{TestHdfsCloudBackupRestore}}. Also, test need to repeat backup/restore
> operation to make sure that closing hdfs filesystem doesn't break it see
> SOLR-9961 for the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13587) Close BackupRepository after every usage
[
https://issues.apache.org/jira/browse/SOLR-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876440#comment-16876440
]
Kevin Risden commented on SOLR-13587:
-
[~mkhludnev] - No I asked around and didn't seem to be any way to work around
the HDFS close stuff. Would need to be fixed in Hadoop first.
> Close BackupRepository after every usage
>
>
> Key: SOLR-13587
> URL: https://issues.apache.org/jira/browse/SOLR-13587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Affects Versions: 8.1
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-13587.patch
>
>
> Turns out BackupRepository is created every operation, but never closed. I
> suppose it leads to necessity to have {{BadHdfsThreadsFilter}} in
> {{TestHdfsCloudBackupRestore}}. Also, test need to repeat backup/restore
> operation to make sure that closing hdfs filesystem doesn't break it see
> SOLR-9961 for the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13587) Close BackupRepository after every usage
[
https://issues.apache.org/jira/browse/SOLR-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876330#comment-16876330
]
Lucene/Solr QA commented on SOLR-13587:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 31m
45s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13587 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12973276/SOLR-13587.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 82bf957 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/475/testReport/ |
| modules | C: solr/core U: solr/core |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/475/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Close BackupRepository after every usage
>
>
> Key: SOLR-13587
> URL: https://issues.apache.org/jira/browse/SOLR-13587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Affects Versions: 8.1
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-13587.patch
>
>
> Turns out BackupRepository is created every operation, but never closed. I
> suppose it leads to necessity to have {{BadHdfsThreadsFilter}} in
> {{TestHdfsCloudBackupRestore}}. Also, test need to repeat backup/restore
> operation to make sure that closing hdfs filesystem doesn't break it see
> SOLR-9961 for the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13587) Close BackupRepository after every usage
[
https://issues.apache.org/jira/browse/SOLR-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876014#comment-16876014
]
Mikhail Khludnev commented on SOLR-13587:
-
[~krisden], Right. However, there's such thing as
[https://issues.apache.org/jira/browse/SOLR-9961?focusedCommentId=15822297=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15822297|unexpected
closing]. Is there any suggestions from HDFS folks in scope of SOLR-5007?
> Close BackupRepository after every usage
>
>
> Key: SOLR-13587
> URL: https://issues.apache.org/jira/browse/SOLR-13587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Affects Versions: 8.1
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-13587.patch
>
>
> Turns out BackupRepository is created every operation, but never closed. I
> suppose it leads to necessity to have {{BadHdfsThreadsFilter}} in
> {{TestHdfsCloudBackupRestore}}. Also, test need to repeat backup/restore
> operation to make sure that closing hdfs filesystem doesn't break it see
> SOLR-9961 for the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13587) Close BackupRepository after every usage
[
https://issues.apache.org/jira/browse/SOLR-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875878#comment-16875878
]
Kevin Risden commented on SOLR-13587:
-
And when I say "can't" I mean literally there is no close option on an HDFS
filesystem instance. It leaks threads since the filesystem instance starts some
thread pools and has no close method to stop them. It would be great if we
could actually call close on an HDFS filesystem, but nope.
> Close BackupRepository after every usage
>
>
> Key: SOLR-13587
> URL: https://issues.apache.org/jira/browse/SOLR-13587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Affects Versions: 8.1
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-13587.patch
>
>
> Turns out BackupRepository is created every operation, but never closed. I
> suppose it leads to necessity to have {{BadHdfsThreadsFilter}} in
> {{TestHdfsCloudBackupRestore}}. Also, test need to repeat backup/restore
> operation to make sure that closing hdfs filesystem doesn't break it see
> SOLR-9961 for the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13587) Close BackupRepository after every usage
[
https://issues.apache.org/jira/browse/SOLR-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875877#comment-16875877
]
Kevin Risden commented on SOLR-13587:
-
Yea so HDFS filesystem instances can't be closed. Sadly. I looked into this as
part of SOLR-5007. I found the same thing that backup repositories don't
open/close things properly. Probably hidden by the fact that HDFS leaks threads.
> Close BackupRepository after every usage
>
>
> Key: SOLR-13587
> URL: https://issues.apache.org/jira/browse/SOLR-13587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Affects Versions: 8.1
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-13587.patch
>
>
> Turns out BackupRepository is created every operation, but never closed. I
> suppose it leads to necessity to have {{BadHdfsThreadsFilter}} in
> {{TestHdfsCloudBackupRestore}}. Also, test need to repeat backup/restore
> operation to make sure that closing hdfs filesystem doesn't break it see
> SOLR-9961 for the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13587) Close BackupRepository after every usage
[
https://issues.apache.org/jira/browse/SOLR-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875476#comment-16875476
]
Mikhail Khludnev commented on SOLR-13587:
-
Turns out even closing repositories doesn't prevent leaking MiniDFSCluster from
leakage..
> Close BackupRepository after every usage
>
>
> Key: SOLR-13587
> URL: https://issues.apache.org/jira/browse/SOLR-13587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Affects Versions: 8.1
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
>
> Turns out BackupRepository is created every operation, but never closed. I
> suppose it leads to necessity to have {{BadHdfsThreadsFilter}} in
> {{TestHdfsCloudBackupRestore}}. Also, test need to repeat backup/restore
> operation to make sure that closing hdfs filesystem doesn't break it see
> SOLR-9961 for the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
