[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752109#comment-16752109 ] Tim Owen commented on SOLR-13029: - Not sure - I can see someone might want parallelised file copies as well, so that ticket is still valid I think. It probably depends on how many collections you have to restore, if (like us) you have many collections to do, we just kick them off in parallel and let each one work through its files in series. But if you had 1 or 2 large collections it might be better done with the proposed change there. > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Assignee: Mikhail Khludnev >Priority: Major > Fix For: 8.0, 7.7, master (9.0) > > Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752082#comment-16752082 ] Mikhail Khludnev commented on SOLR-13029: - [~TimOwen] does it mean SOLR-9961 is unlocked? > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Assignee: Mikhail Khludnev >Priority: Major > Fix For: 8.0, 7.7, master (9.0) > > Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752062#comment-16752062 ] Tim Owen commented on SOLR-13029: - Thanks Mikhail! > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Assignee: Mikhail Khludnev >Priority: Major > Fix For: 8.0, 7.7, master (9.0) > > Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751281#comment-16751281 ] ASF subversion and git services commented on SOLR-13029: Commit 5a54c624cac4d793c7e65c980eccde3c2d63e61b in lucene-solr's branch refs/heads/branch_8x from Mikhail Khludnev [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5a54c62 ] SOLR-13029: configure buffer size in HdfsBackupRepository. > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751280#comment-16751280 ] ASF subversion and git services commented on SOLR-13029: Commit c1ce24e0cfbd40aacc3c0f0117d9be56cc1b7936 in lucene-solr's branch refs/heads/branch_7x from Mikhail Khludnev [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c1ce24e ] SOLR-13029: configure buffer size in HdfsBackupRepository. > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751270#comment-16751270 ] ASF subversion and git services commented on SOLR-13029: Commit c317119654c600f89eb29a423c51a8029f84033d in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c317119 ] SOLR-13029: configure buffer size in HdfsBackupRepository. > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[
https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751032#comment-16751032
]
Mikhail Khludnev commented on SOLR-13029:
-
I like the patch, going to commit soon. Failed {{TestSimExtremeIndexing}}
doesn't seem relevant.
> Allow HDFS backup/restore buffer size to be configured
> --
>
> Key: SOLR-13029
> URL: https://issues.apache.org/jira/browse/SOLR-13029
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore, hdfs
>Affects Versions: 7.5, 8.0
>Reporter: Tim Owen
>Priority: Major
> Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch
>
>
> There's a default hardcoded buffer size setting of 4096 in the HDFS code
> which means in particular that restoring a backup from HDFS takes a long
> time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes
> is very inefficient. We changed this in our local build used in production to
> 256kB and saw a 10x speed improvement when restoring a backup. Attached patch
> simply makes this size configurable using a command line setting, much like
> several other buffer size values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[
https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750953#comment-16750953
]
Lucene/Solr QA commented on SOLR-13029:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 2m 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 2m 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 2m 5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 54s{color}
| {color:red} core in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | solr.cloud.autoscaling.sim.TestSimExtremeIndexing |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13029 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955939/SOLR-13029.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene2-us-west.apache.org 4.4.0-112-generic #135-Ubuntu SMP
Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 72a99e9 |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-SOLR-Build/266/artifact/out/patch-unit-solr_core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/266/testReport/ |
| modules | C: solr/core U: solr/core |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/266/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Allow HDFS backup/restore buffer size to be configured
> --
>
> Key: SOLR-13029
> URL: https://issues.apache.org/jira/browse/SOLR-13029
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore, hdfs
>Affects Versions: 7.5, 8.0
>Reporter: Tim Owen
>Priority: Major
> Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch
>
>
> There's a default hardcoded buffer size setting of 4096 in the HDFS code
> which means in particular that restoring a backup from HDFS takes a long
> time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes
> is very inefficient. We changed this in our local build used in production to
> 256kB and saw a 10x speed improvement when restoring a backup. Attached patch
> simply makes this size configurable using a command line setting, much like
> several other buffer size values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749806#comment-16749806 ] Tim Owen commented on SOLR-13029: - hah, I wasn't suggesting automating that.. just how I manually tested it. I've attached a newer patch, containing some unit tests for the various situations > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747234#comment-16747234 ] Mikhail Khludnev commented on SOLR-13029: - bq. I used a heap dump to confirm that the buffer really was the size I set in the configuration. I don't think we can afford it with Jenkins. > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746174#comment-16746174 ] Tim Owen commented on SOLR-13029: - Sure - there's not a huge amount of code logic paths to test, but I can take a look. In practice, I used a heap dump to confirm that the buffer really was the size I set in the configuration. > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745948#comment-16745948 ] Mikhail Khludnev commented on SOLR-13029: - Just an idea for test coverage, we can inject illegal numbers via config and assert code failure. WDYT? > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, 8.0 >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13029) Allow HDFS backup/restore buffer size to be configured
[ https://issues.apache.org/jira/browse/SOLR-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729614#comment-16729614 ] Mikhail Khludnev commented on SOLR-13029: - [~varunthacker], [~hgadre], what's your take on that? > Allow HDFS backup/restore buffer size to be configured > -- > > Key: SOLR-13029 > URL: https://issues.apache.org/jira/browse/SOLR-13029 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, hdfs >Affects Versions: 7.5, master (8.0) >Reporter: Tim Owen >Priority: Major > Attachments: SOLR-13029.patch, SOLR-13029.patch > > > There's a default hardcoded buffer size setting of 4096 in the HDFS code > which means in particular that restoring a backup from HDFS takes a long > time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes > is very inefficient. We changed this in our local build used in production to > 256kB and saw a 10x speed improvement when restoring a backup. Attached patch > simply makes this size configurable using a command line setting, much like > several other buffer size values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
