[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555440#comment-16555440
]
ASF subversion and git services commented on LUCENE-8415:
-
Commit 608f0277b07c73218e5dc1d8ce3a264020c50f36 in lucene-solr's branch
refs/heads/branch_7x from [~dawid.weiss]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=608f027 ]
LUCENE-8415: test quirk follow up. MockDirectoryWriter uses
AccessDeniedException (a subclass of IOException) to signal files still open
for writing when read access is requested.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Attachments: LUCENE-8415.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555438#comment-16555438
]
ASF subversion and git services commented on LUCENE-8415:
-
Commit d25f62634b95e49df739a5a2612f9b719ef3a77d in lucene-solr's branch
refs/heads/master from [~dawid.weiss]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d25f626 ]
LUCENE-8415: test quirk follow up. MockDirectoryWriter uses
AccessDeniedException (a subclass of IOException) to signal files still open
for writing when read access is requested.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Attachments: LUCENE-8415.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555364#comment-16555364
]
ASF subversion and git services commented on LUCENE-8415:
-
Commit 8892c0d9aff9e8b7a6722a50a18908ef575276da in lucene-solr's branch
refs/heads/jira/http2 from [~dawid.weiss]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8892c0d ]
LUCENE-8415: Clean up Directory contracts (write-once, no
reads-before-write-completed). Minor test improvements and cleanups.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Attachments: LUCENE-8415.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553889#comment-16553889
]
ASF subversion and git services commented on LUCENE-8415:
-
Commit 9dc6cff38a2de26a249659d1dc8240aa716b674c in lucene-solr's branch
refs/heads/branch_7x from [~dawid.weiss]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9dc6cff ]
LUCENE-8415: Clean up Directory contracts (write-once, no
reads-before-write-completed). Minor test improvements and cleanups.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Attachments: LUCENE-8415.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553887#comment-16553887
]
ASF subversion and git services commented on LUCENE-8415:
-
Commit 8892c0d9aff9e8b7a6722a50a18908ef575276da in lucene-solr's branch
refs/heads/master from [~dawid.weiss]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8892c0d ]
LUCENE-8415: Clean up Directory contracts (write-once, no
reads-before-write-completed). Minor test improvements and cleanups.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Attachments: LUCENE-8415.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552710#comment-16552710
]
Dawid Weiss commented on LUCENE-8415:
-
I think it's ready. Passes precommit and tests (repeated 20 times). The patch
contains minor changes to tests -- I cleaned up a few places bringing them to
Java 8 (try-with-resources to close test directories, streams here and there,
hardcoded ExtraFS file checks).
I'll follow-up with ram directory cleanups in a separate patch.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Attachments: LUCENE-8415.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551298#comment-16551298
]
Robert Muir commented on LUCENE-8415:
-
I guess what I mean is, RAMDirectory is different because it lives in a bubble.
On the other hand FSDirectory shares a world with other applications (maybe
java, maybe not) and maybe even other computers in more ridiculous setups. So
java code in the Directory isn't really up for the task of enforcing, we need
OS help.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551227#comment-16551227
]
Dawid Weiss commented on LUCENE-8415:
-
Yep, I agree. Doesn't make sense to make FS impls. slower just to enforce it,
it's enough that we run tests that capture it early. I'll work on it.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551224#comment-16551224
]
Robert Muir commented on LUCENE-8415:
-
I don't think such stuff belongs in the directory. We should be leaning on the
operating system for such guarantees.
I know you've been looking at RAMDirectory, but its really an atypical
case/wildcard.
MockDirectoryWrapper/mockfs stuff should have all the assertions we can throw
at it.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551188#comment-16551188
]
Dawid Weiss commented on LUCENE-8415:
-
Oh, I absolutely agree. The thing is should we try to enforce it at runtime, in
the code of each directory, or only verify that we don't do it in the tests (in
MockDirectoryWrapper)?
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551147#comment-16551147
]
Robert Muir commented on LUCENE-8415:
-
by this i mean, use atomic rename for all index files not just segments_N. then
nobody can be reading from them until they are complete.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551141#comment-16551141
]
Dawid Weiss commented on LUCENE-8415:
-
bq. you could also do no bookkeeping and simply pay the cost of more renames,
right?
Err, I don't follow? My idea was to ensure a file open for output cannot be
open for input (until the output is closed). This proves quite extensive
code-wise to enforce, so I think it'll be better to leave it up to the
implementation to decide (Directory.openInput may throw an exception on an
input that is still being written to). I'll update the patch and submit for
review (on Monday, unfortunately -- I'm away for the weekend).
On a positive note, removing segment* exceptions from the mock classes didn't
break anything after beasting overnight.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550740#comment-16550740
]
Robert Muir commented on LUCENE-8415:
-
you could also do no bookkeeping and simply pay the cost of more renames,
right? Currently only the segments_N is written atomically like this. But
writing to a temp file and then renaming at the end is pretty easy to
understand, lots of applications do it.
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8415) Clean up Directory contracts (write-once, no reads-before-write-completed)
[
https://issues.apache.org/jira/browse/LUCENE-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550614#comment-16550614
]
Dawid Weiss commented on LUCENE-8415:
-
Ensuring {{testReadFileOpenForWrites}} works would require moving some
bookkeeping to Directory classes (and IndexOutput implementations). A
concurrent hash map of open outputs and an update on IndexOutput.close(),
essentially.
We have a few options.
Make it a contractual requirement (then we have to implement this bookkeeping
for true filesystems since they allow readers over a writer for the same
process). Make this an assertion-mode only check (implement book keeping, but
don't run it except for assertion-enabled runs). Finally, don't make any checks
at all, but give the contractual liberty for Directory implementations to throw
AccessDeniedException in {{openInput}} if a file is still open.
The offending directory implementations right now are:
{code}
- org.apache.lucene.store.TestTrackingDirectoryWrapper.testReadFileOpenForWrites
[junit4] -
org.apache.lucene.store.TestMmapDirectory.testReadFileOpenForWrites
[junit4] -
org.apache.lucene.store.TestSimpleFSDirectory.testReadFileOpenForWrites
[junit4] -
org.apache.lucene.store.TestNRTCachingDirectory.testReadFileOpenForWrites
[junit4] -
org.apache.lucene.store.TestFileSwitchDirectory.testReadFileOpenForWrites
[junit4] -
org.apache.lucene.store.TestRAMDirectory.testReadFileOpenForWrites
[junit4] -
org.apache.lucene.store.TestNIOFSDirectory.testReadFileOpenForWrites
[junit4] - org.apache.lucene.store.TestMultiMMap.testReadFileOpenForWrites
[junit4] -
org.apache.lucene.store.TestFilterDirectory.testReadFileOpenForWrites
{code}
> Clean up Directory contracts (write-once, no reads-before-write-completed)
> --
>
> Key: LUCENE-8415
> URL: https://issues.apache.org/jira/browse/LUCENE-8415
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Created a PR here for early review.
> https://github.com/apache/lucene-solr/pull/424
> I changed:
> * the wording in Directory documentation to be a bit more formalized about
> what rules a Directory should obey (and users expect).
> * modified the test framework to verify the above in mock classes.
> Currently a number of Directory implementations fail the
> {{testReadFileOpenForWrites}} test that I added, so I'll keep working on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
