Chris Hostetter wrote:
Compression of stored fields is a feature that the Lucene "core"
currently
supports out of the box -- but it does so in a very limited maner
that
doesn't allow for much configuration. There is no advantage for
users in
using compressed fields over compressing the data themselves
before adding
it to the index, only disdvantages: notably the limited control
the user
has over the compression, and added complexity for the code path
executed
by all users -- even if they don't use compression (a boolean test on
"compressed" in FieldsReader may be fast ... but it's still a
bytecode op
for every field that's completley uneccessary for a large portion
of the
user base)
If the code was not already in the core, and someone asked about
adding it
I would argue against doing so on the grounds that some helpfull
utility
methods (possibly in a contrib) would be just as usefull, and
would have
no performance cost for people who don't care about compression.
Perhaps, if you look at compression on its own, but once you see
compression
in the context of all the other field options it makes sense to
have it
added to Lucene, it's about having everything in one place for ease of
implementation that offsets the performance issue, in my opinion.
First off, if all we are interested in in Encrypting *stored* data,
then the issue becomes exactly the same as compression: there is
no point
in putting this functionality in the "core" Lucene code base when
it can
be done using helper utility methods -- now that that's out of the
way,
let's talk about the good stuff...
As above
If we want to encrypt the text portion of Terms that are index for a
specific set of fields, this is again something that can easily be
done
without modifying the "core" Lucene code base -- utility methods
can be
used to help people encrypt UN_TOKENIZED Field values, and a simple
AnalyzerWrapper can be made to encrypt the text portion of Tokens
produced
by another analyzer both when indexing Field values and when
QueryParser
is Analyzing input text if neccessary.
I take your word for it, but wouldn't you agree that replacing all
the above
with just one line, "Field.Store.Encrypted" (or
Field.Store.Encrypt, for
compatibility with Field.Store.Compress),would be a lot easier to
use for
the average developer?
As others have already pointed out: encrypting just the Term text
doesn't
do much to aid the overall security of your data -- because a bad
guy with
access to your index can use the various statistics about your terms
(docFreq, term vectors, term positions, etc...) to aid them in
cracking
your encryption -- maybe a user is okay with that risk, in which
case my
previous comment about how this can easily be done without
modifying any
core lucene classes still holds. what about users who don't think
this is
an acceptible risk? ... a more robust encryption mechanism is
neccessary...
Security is a big topic, we cannot hope to discuss it here. I am
talking
about some form of data protection, not security.
When you say "a bad guy with access to your index", you imply that
nothing
can be done to protect the index. But accessing an index which you are
determined to protect would not be easy, would require expertise,
money, as
well as the risk of a potential jail sentence. If you have National
Security
in mind, be assured no agency responsible for national security
will use
open source software which is not certified, and that is downloaded
from an
unsecure site over the internet, in order to protect the nation (I
hope!).
If we are talking about applications which need to protect data
from curious
or even ill-intentioned eyes, then you can provide a deterrent by
encrypting
that sensitive data only. It might be a list of names, or balances, or
credit card numbers. Lucene alone can only provide some form of data
protection, not security. If you accept this limitation you will
find it
easier to accept the notion of encryption at field level, just like
some
relational database software encrypts at column level. Just as
importantly
you want to be able to search over that encrypted field, somehing
which my
proposed code provides (within the stated current limitation).
So exactly what pieces of data about a set of fields in an index
need to
be encrypted before you can adequetly say that those fields are
encrypted?
Off the top of my head i don't know, but I think the only way to
play it
safe is to assume thta *all* of the data needs to be encrypted.
Cannot agree here, it's application dependent. And keep in mind
that once
you offer new functionality people will find many original
applications for
it.
Now the question becomes: do we modify all of the index writitng/
reading
code
to add a lot of "if (encrypted) { ... } else { ... }" checks, or
is there
an easier way to ensure that all of the data in encrypted without
impacting the majority of hte user base?
A perfectly valid point, only benchmarking will tell by how much
the current
performance of Lucene will be impacted by the addition of encryption.
Somebody in this discussion suggested a Lucene benchmarking tool
which can
be used. I am not familiar with it, but if it is easy to run then
let's do
it and resolve factually this part of the discussion.
On a more philosophical level, are you saying that there should not
be any
added functionality to Lucene if it impacts the performance of
those who do
not need the additional functionality. This could be a major
limitation to
the future of Lucene. Perhaps one should set some small % limits to
the
level of impact, but zero could be too limiting.
I would argue that creating an EncryptedDirectory class with an
API that
looks something like this.......
.............
.............
- Do my concerns about that impact make sense to you?
- Does my (high level) description of how i think encryption
might make
sense as an optional Lucene feature make sense?
- are there any advantages you see to your approach that you feel
make it
more worthwhile then a Directory based approach?
Points one and two are pefectly valid and make a lot of sense.
Point three
is about what is best for the most, given that there is already an
OS option
to encrypt at directory level.
I like field encryption because it is functionality which cannot be
implemented at the OS level, and because of its granularity and its
similarity to existing Lucene functionality, it would be more
intuitive and
easier to implement at the application level. Encrypting everything
in a
directory would have a performance impact on the application.
I accept your point about the difference between a file system
directory and
a Lucene directory. But in order to overcome the lack of field-level
encryption and to minimise the performance impact on the
application you
would be forced to create a separate index and directory for each
field
which you want encrypted. It will work, but is not a solution I
would like
to have adopt at the application level.
Finally a point about my code. I was unsuccessful in creating a
diff file
because I was picking up all kind of formatting differences as
well. If you
scan it quickly you will find that is really very simple and, at
least in
its current limited implementation, hardly invasive of Lucene's
core. All
the encryption routines are in a separate class which i placed in the
utility package.
Victor
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
View this message in context: http://www.nabble.com/Attached-
proposed-modifications-to-Lucene-2.0-to-support-
Field.Store.Encrypted-tf2727614.html#a7708481
Sent from the Lucene - Java Developer mailing list archive at
Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
