Build failed in Jenkins: pirk » Apache Pirk (incubating) Project #46

2016-09-20 Thread Apache Jenkins Server 
See 

--
Established TCP socket on 53965
maven3-agent.jar already up to date
maven3-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 
 
-Dmaven.repo.local=/home/jenkins/jenkins-slave/maven-repositories/0 install
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building Apache Pirk (incubating) Project 0.2.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- apache-rat-plugin:0.12:check (default) @ apache-pirk ---
[INFO] Enabled default license matchers.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Parsing exclusions from 

[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 74 implicit excludes (use -debug for more details).
[INFO] Exclude: .travis.yml
[INFO] Exclude: findbugs-exclude.xml
[INFO] Exclude: KEYS
[INFO] Exclude: eclipse*.xml
[INFO] Exclude: docs/*
[INFO] Exclude: logs/*
[INFO] Exclude: **/m2.conf
[INFO] Exclude: src/main/resources/META-INF/**
[INFO] 549 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 1, unknown: 1, generated: 
0, approved: 123 licenses.

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

2016-09-20 Thread Ellison Anne Williams 
I am in favor of breaking out pirk-core as specified so that our initial
submodule structure would be as follows:

pirk-core (encryption,query, inputformat, serialization, utils)

pirk-responder (core responder incl. standalone)

pirk-querier

pirk-storm

pirk-mapreduce

pirk-spark

pirk-benchmark

pirk-distributed-test


One thing to note is that under this breakdown, pirk-core would not include
the Elasticsearch dependency (es-hadoop). The only submodules that would
have the es-hadoop dependency (those which need it) currently are
pirk-mapreduce, pirk-spark, and pirk-distributed-test.


I believe that we agreed (somewhere :)) in this thread to go ahead and
remove the platform 'backwards compatibility' for PIRK-63. Please holler if
this is not correct.




On Tue, Sep 20, 2016 at 9:40 PM, Darin Johnson 
wrote:

> Suneel, a google doc as promised, only a day late (sorry - sick kid).
>
> https://docs.google.com/document/d/1K8E0TridC1hBfqDwWCqdZ8Dj_5_
> mMrRQyynQ-Q6MFbI/edit?usp=sharing
>
> I was planning on working on this, but I'm going to take a day or two to
> let others comment.
>
> Darin
>
> On Mon, Sep 19, 2016 at 5:07 PM, Suneel Marthi 
> wrote:
>
> > A shared Google doc would be more convenient than a bunch of Jiras. Its
> > easier to comment and add notes that way.
> >
> >
> > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson  >
> > wrote:
> >
> > > Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
> > > Based off my pirk-63 I was able to pull spark and storm out with no
> > > issues.  I was planning to pull them out, then tackling elastic search,
> > > then hadoop as it's a little entrenched.  This should keep most PRs to
> > > manageable chunks. I think once that's done addressing the configs will
> > > make more sense.
> > >
> > > I'm open to suggestions. But the hope would be:
> > > Pirk-parent
> > > Pirk-core
> > > Pirk-hadoop
> > > Pirk-storm
> > > Pirk-parent
> > >
> > > Pirk-es is a little weird as it's really just an inputformat, seems
> like
> > > there's a more general solution here than creating submodules for every
> > > inputformat.
> > >
> > > Darin
> > >
> > > On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
> > >
> > > >
> > >
> > > > Refactor is definitely a first priority.  Is there a design/proposal
> > > draft
> > > > that we could comment on about how to go about refactoring the
> code.  I
> > > > have been trying to keep up with the emails but definitely would have
> > > > missed some.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > > > eawilli...@apache.org > wrote:
> > > >
> > > > > Agree - let's leave the config/CLI the way it is for now and tackle
> > > that as
> > > > > a subsequent design discussion and PR.
> > > > >
> > > > > Also, I think that we should leave the ResponderDriver and the
> > > > > ResponderProps alone for this PR and push to a subsequent PR (once
> we
> > > > > decide if and how we would like to delegate each).
> > > > >
> > > > > I vote to remove the 'platform' option and the backwards
> > compatibility
> > > in
> > > > > this PR and proceed with having a ResponderLauncher interface and
> > > forcing
> > > > > its implementation by the ResponderDriver.
> > > > >
> > > > > And, I am not so concerned with having one fat jar vs. multiple
> jars
> > > right
> > > > > now - to me, at this point, it's a 'nice to have' and not a 'must
> > have'
> > > for
> > > > > Pirk functionality. We do need to break out Pirk into more clearly
> > > defined
> > > > > submodules (which is in progress) - via this re-factor, I think
> that
> > we
> > > > > will gain some ability to generate multiple jars which is nice.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <
> t.p.elli...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > On 19/09/16 15:46, Darin Johnson wrote:
> > > > > > > Hey guys,
> > > > > > >
> > > > > > > Thanks for looking at the PR, I apologize if it offended
> anyone's
> > > > > eyes:).
> > > > > > >
> > > > > > > I'm glad it generated some discussion about the
> configuration.  I
> > > > > didn't
> > > > > > > really like where things were heading with the config.
> However,
> > > didn't
> > > > > > > want to create to much scope creep.
> > > > > > >
> > > > > > > I think any hierarchical config (TypeSafe or yaml) would make
> > > things
> > > > > much
> > > > > > > more maintainable, the plugin could simply grab the appropriate
> > > part of
> > > > > > the
> > > > > > > config and handle accordingly.  I'd also cut down the number of
> > > command
> > > > > > > line options to only those that change between runs often (like
> > > > > > > input/output)
> > > > > > >
> > > > > > >> One option is to make Pirk pluggable, so that a Pirk
> > installation
> > > > > could
> > > > > > >> use one or more of these in an extensible fashion by

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

2016-09-20 Thread Darin Johnson 
Suneel, a google doc as promised, only a day late (sorry - sick kid).

https://docs.google.com/document/d/1K8E0TridC1hBfqDwWCqdZ8Dj_5_mMrRQyynQ-Q6MFbI/edit?usp=sharing

I was planning on working on this, but I'm going to take a day or two to
let others comment.

Darin

On Mon, Sep 19, 2016 at 5:07 PM, Suneel Marthi 
wrote:

> A shared Google doc would be more convenient than a bunch of Jiras. Its
> easier to comment and add notes that way.
>
>
> On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson 
> wrote:
>
> > Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
> > Based off my pirk-63 I was able to pull spark and storm out with no
> > issues.  I was planning to pull them out, then tackling elastic search,
> > then hadoop as it's a little entrenched.  This should keep most PRs to
> > manageable chunks. I think once that's done addressing the configs will
> > make more sense.
> >
> > I'm open to suggestions. But the hope would be:
> > Pirk-parent
> > Pirk-core
> > Pirk-hadoop
> > Pirk-storm
> > Pirk-parent
> >
> > Pirk-es is a little weird as it's really just an inputformat, seems like
> > there's a more general solution here than creating submodules for every
> > inputformat.
> >
> > Darin
> >
> > On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
> >
> > >
> >
> > > Refactor is definitely a first priority.  Is there a design/proposal
> > draft
> > > that we could comment on about how to go about refactoring the code.  I
> > > have been trying to keep up with the emails but definitely would have
> > > missed some.
> > >
> > >
> > >
> > > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > > eawilli...@apache.org > wrote:
> > >
> > > > Agree - let's leave the config/CLI the way it is for now and tackle
> > that as
> > > > a subsequent design discussion and PR.
> > > >
> > > > Also, I think that we should leave the ResponderDriver and the
> > > > ResponderProps alone for this PR and push to a subsequent PR (once we
> > > > decide if and how we would like to delegate each).
> > > >
> > > > I vote to remove the 'platform' option and the backwards
> compatibility
> > in
> > > > this PR and proceed with having a ResponderLauncher interface and
> > forcing
> > > > its implementation by the ResponderDriver.
> > > >
> > > > And, I am not so concerned with having one fat jar vs. multiple jars
> > right
> > > > now - to me, at this point, it's a 'nice to have' and not a 'must
> have'
> > for
> > > > Pirk functionality. We do need to break out Pirk into more clearly
> > defined
> > > > submodules (which is in progress) - via this re-factor, I think that
> we
> > > > will gain some ability to generate multiple jars which is nice.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison  >
> > > > wrote:
> > > >
> > > > > On 19/09/16 15:46, Darin Johnson wrote:
> > > > > > Hey guys,
> > > > > >
> > > > > > Thanks for looking at the PR, I apologize if it offended anyone's
> > > > eyes:).
> > > > > >
> > > > > > I'm glad it generated some discussion about the configuration.  I
> > > > didn't
> > > > > > really like where things were heading with the config.  However,
> > didn't
> > > > > > want to create to much scope creep.
> > > > > >
> > > > > > I think any hierarchical config (TypeSafe or yaml) would make
> > things
> > > > much
> > > > > > more maintainable, the plugin could simply grab the appropriate
> > part of
> > > > > the
> > > > > > config and handle accordingly.  I'd also cut down the number of
> > command
> > > > > > line options to only those that change between runs often (like
> > > > > > input/output)
> > > > > >
> > > > > >> One option is to make Pirk pluggable, so that a Pirk
> installation
> > > > could
> > > > > >> use one or more of these in an extensible fashion by adding JAR
> > files.
> > > > > >> That would still require selecting one by command-line argument.
> > > > > >
> > > > > > An argument for this approach is for lambda architecture
> approaches
> > > > (say
> > > > > > spark/spark-streaming) were the contents of the jars would be so
> > > > similar
> > > > > it
> > > > > > seems like to much trouble to create separate jars.
> > > > > >
> > > > > > Happy to continue working on this given some direction on where
> > you'd
> > > > > like
> > > > > > it to go.  Also, it's a bit of a blocker to refactoring the build
> > into
> > > > > > submodules.
> > > > >
> > > > > FWIW my 2c is to not try and fix all the problems in one go, and
> > rather
> > > > > take a compromise on the configurations while you tease apart the
> > > > > submodules in to separate source code trees, poms, etc; then come
> > back
> > > > > and fix the runtime configs.
> > > > >
> > > > > Once the submodules are in place it will open up more work for
> > release
> > > > > engineering and tinkering that can be done in parallel with the
> > config
> > > > >

Re: Math deck

2016-09-20 Thread Walter Ray-Dulany 
> Please can you clarify, the doc for Wideskies algorithm (slide 16) says that
zeta is chosen to be in (Z/N^2 Z)* -- but the code we have in Paillier.java[1]
appears to be selecting for (Z/NZ)*

Yes! Slide 22 (formerly slide 16) is wrong; zeta is in (Z/NZ)*. I, or
someone else, can update the slides.

Good catch.

On Tue, Sep 20, 2016 at 4:10 PM, Ellison Anne Williams <
eawilliamsp...@gmail.com> wrote:

> On Tue, Sep 20, 2016 at 12:10 PM, Tim Ellison 
> wrote:
>
> > On 20/09/16 15:20, Walter Ray-Dulany wrote:
> > >> Is this the same as I've seen this written elsewhere as double
> stroked Z
> > >> subscript N?
> > >
> > > Most definitely. I write it the way that I do to for historical reasons
> > > (mathematical and personal).
> >
> > Ok, works well in ascii comments too.
> >
> > >> I assume that, practically, it is better to work with smaller factors
> > to make
> > >> the math operations faster (but not too small to compromise security)
> > and
> > >> chunk the message accordingly, rather than working with large
> messages.
> > >
> > > This is precisely what Pirk does.
> >
> > I guess my question is, does Pirk change the message chunk size
> > depending upon the bit length of N?  Seems that there is going to be
> > some correlation there - if the message length is too small you are
> > doing more calculations than necessary.
> >
> > Is this chunking the DataPartitioner's job?  In which case I see the
> > built-in partitioners are all going to 8-bit partitions.
> >
>
> No - just to be clear - the message size is independent of the modulus size
> (N). The message is broken up into 'chunks' (currently 8 bits in length)
> and then 'striped' across the rows of the matrix in order to perform the
> encrypted query. This data 'chunking' is the responsibility of the
> DataPartitioner in Pirk. There are other dependencies with N in the
> parameter selection, but the message size is not one of them.
>
>
>
> > >> Whoa, why is g = 1 + N the standard Wideskies g?
> > >
> > > (1) It works (as I showed above), and (2) it turns out that all such g
> > are
> > > powers of (1+N). I'm not going to prove that, but it's true.
> > >
> > > Paillier's paper goes into detail about how the security of the system
> is
> > > independent of the g one chooses. Since g is public, and since the
> > security
> > > of the system does not suffer for just choosing the same g every time,
> > > Wideskies just says "screw it, we're going with 1+N".
> >
> > I'm curious, but I'll draw the line there and trust :-)
> >
> > >> How do you want to receive proposed changes to the PDF, if at all?
> I'm
> > rephrasing
> > >> your version a bit as I go, and quite happy to keep it separate so
> > there is
> > >> a simple version for those that don't "speak algebra".
> > >
> > > I've created another issue, PIRK-70, to add the LaTeX beamer doc from
> > which
> > > I generated the pdf, along with the relevant Pirk images. These docs
> will
> > > live in the main branch (not the web branch) under contrib. This is in
> > > addition to the changes I've made to the PDF to clarify the math
> > language.
> > > With the TeX there, anyone can edit the doc, if they so choose. Tim,
> you
> > > could do that, or roll your own in whatever format you like.
> >
> > Cool, thanks.  I'll keep working my way through it as I get a few mins.
> >
> > Please can you clarify, the doc for Wideskies algorithm (slide 16) says
> > that zeta is chosen to be in (Z/N^2 Z)* -- but the code we have in
> > Paillier.java[1] appears to be selecting for (Z/NZ)*
> >
> > [1]
> > https://github.com/apache/incubator-pirk/blob/master/
> > src/main/java/org/apache/pirk/encryption/Paillier.java#L257
> >
> > Regards,
> > Tim
> >
> > > What do you think?
> > >
> > > Walter
> > >
> > > On Tue, Sep 20, 2016 at 8:33 AM, Tim Ellison 
> > wrote:
> > >
> > >> On 19/09/16 18:36, Walter Ray-Dulany wrote:
> > >>> Apologies! It's disorienting at first, and most of all when one
> > actually
> > >>> tries to sit down and do a real example. The version on the slides
> was
> > >> not
> > >>> written in one go, I assure you.
> > >>>
> > >>> Let's go through, and see what's not working.
> > >>
> > >> I appreciate your patience Walter.  I'm sure this "just makes sense"
> to
> > >> you, but my brain is hurting trying to keep up :-)
> > >>
> > >>> **
> > >>>
> >  I'm trying a very simple example.  I'm going to choose, p = 3, q = 5
> > and
> > >>> a message m = 42
> > >>>
> > >>> Already we're in trouble. p and q are fine; but remember that the
> > >> plaintext
> > >>> space (let's call it P(N)) is the set of all integers in Z/NZ; that
> is,
> > >> it
> > >>> is all numbers m
> > >>>
> > >>> 0 <=  m < N
> > >>
> > >> Ah, so understanding the notation was confusing me, which is not going
> > >> to help!  Is this the same as I've seen this written elsewhere as
> double
> > >> stroked Z subscript N?
> > >>
> > >>> You can see already that the m you

Re: Math deck

2016-09-20 Thread Ellison Anne Williams 
On Tue, Sep 20, 2016 at 12:10 PM, Tim Ellison  wrote:

> On 20/09/16 15:20, Walter Ray-Dulany wrote:
> >> Is this the same as I've seen this written elsewhere as double stroked Z
> >> subscript N?
> >
> > Most definitely. I write it the way that I do to for historical reasons
> > (mathematical and personal).
>
> Ok, works well in ascii comments too.
>
> >> I assume that, practically, it is better to work with smaller factors
> to make
> >> the math operations faster (but not too small to compromise security)
> and
> >> chunk the message accordingly, rather than working with large messages.
> >
> > This is precisely what Pirk does.
>
> I guess my question is, does Pirk change the message chunk size
> depending upon the bit length of N?  Seems that there is going to be
> some correlation there - if the message length is too small you are
> doing more calculations than necessary.
>
> Is this chunking the DataPartitioner's job?  In which case I see the
> built-in partitioners are all going to 8-bit partitions.
>

No - just to be clear - the message size is independent of the modulus size
(N). The message is broken up into 'chunks' (currently 8 bits in length)
and then 'striped' across the rows of the matrix in order to perform the
encrypted query. This data 'chunking' is the responsibility of the
DataPartitioner in Pirk. There are other dependencies with N in the
parameter selection, but the message size is not one of them.



> >> Whoa, why is g = 1 + N the standard Wideskies g?
> >
> > (1) It works (as I showed above), and (2) it turns out that all such g
> are
> > powers of (1+N). I'm not going to prove that, but it's true.
> >
> > Paillier's paper goes into detail about how the security of the system is
> > independent of the g one chooses. Since g is public, and since the
> security
> > of the system does not suffer for just choosing the same g every time,
> > Wideskies just says "screw it, we're going with 1+N".
>
> I'm curious, but I'll draw the line there and trust :-)
>
> >> How do you want to receive proposed changes to the PDF, if at all?  I'm
> rephrasing
> >> your version a bit as I go, and quite happy to keep it separate so
> there is
> >> a simple version for those that don't "speak algebra".
> >
> > I've created another issue, PIRK-70, to add the LaTeX beamer doc from
> which
> > I generated the pdf, along with the relevant Pirk images. These docs will
> > live in the main branch (not the web branch) under contrib. This is in
> > addition to the changes I've made to the PDF to clarify the math
> language.
> > With the TeX there, anyone can edit the doc, if they so choose. Tim, you
> > could do that, or roll your own in whatever format you like.
>
> Cool, thanks.  I'll keep working my way through it as I get a few mins.
>
> Please can you clarify, the doc for Wideskies algorithm (slide 16) says
> that zeta is chosen to be in (Z/N^2 Z)* -- but the code we have in
> Paillier.java[1] appears to be selecting for (Z/NZ)*
>
> [1]
> https://github.com/apache/incubator-pirk/blob/master/
> src/main/java/org/apache/pirk/encryption/Paillier.java#L257
>
> Regards,
> Tim
>
> > What do you think?
> >
> > Walter
> >
> > On Tue, Sep 20, 2016 at 8:33 AM, Tim Ellison 
> wrote:
> >
> >> On 19/09/16 18:36, Walter Ray-Dulany wrote:
> >>> Apologies! It's disorienting at first, and most of all when one
> actually
> >>> tries to sit down and do a real example. The version on the slides was
> >> not
> >>> written in one go, I assure you.
> >>>
> >>> Let's go through, and see what's not working.
> >>
> >> I appreciate your patience Walter.  I'm sure this "just makes sense" to
> >> you, but my brain is hurting trying to keep up :-)
> >>
> >>> **
> >>>
>  I'm trying a very simple example.  I'm going to choose, p = 3, q = 5
> and
> >>> a message m = 42
> >>>
> >>> Already we're in trouble. p and q are fine; but remember that the
> >> plaintext
> >>> space (let's call it P(N)) is the set of all integers in Z/NZ; that is,
> >> it
> >>> is all numbers m
> >>>
> >>> 0 <=  m < N
> >>
> >> Ah, so understanding the notation was confusing me, which is not going
> >> to help!  Is this the same as I've seen this written elsewhere as double
> >> stroked Z subscript N?
> >>
> >>> You can see already that the m you chose is not in the plaintext space.
> >>>
> >>> Let's pick a new m to continue with; in this case, let's choose your m,
> >> but
> >>> mod 15 so that it lies in P(N). Thus, our new m going forward shall be
> >>>
> >>> m = 12
> >>
> >> Interesting.  That means that the size of message I can encode directly
> >> using this scheme is related to the bit length of my RSA factors.
> >>
> >> I assume that, practically, it is better to work with smaller factors to
> >> make the math operations faster (but not too small to compromise
> >> security) and chunk the message accordingly, rather than working with
> >> large messages.
> >>
> >>>

Re: Math deck

2016-09-20 Thread Tim Ellison 
On 20/09/16 15:20, Walter Ray-Dulany wrote:
>> Is this the same as I've seen this written elsewhere as double stroked Z
>> subscript N?
> 
> Most definitely. I write it the way that I do to for historical reasons
> (mathematical and personal).

Ok, works well in ascii comments too.

>> I assume that, practically, it is better to work with smaller factors to make
>> the math operations faster (but not too small to compromise security) and
>> chunk the message accordingly, rather than working with large messages.
> 
> This is precisely what Pirk does.

I guess my question is, does Pirk change the message chunk size
depending upon the bit length of N?  Seems that there is going to be
some correlation there - if the message length is too small you are
doing more calculations than necessary.

Is this chunking the DataPartitioner's job?  In which case I see the
built-in partitioners are all going to 8-bit partitions.

>> Whoa, why is g = 1 + N the standard Wideskies g?
> 
> (1) It works (as I showed above), and (2) it turns out that all such g are
> powers of (1+N). I'm not going to prove that, but it's true.
> 
> Paillier's paper goes into detail about how the security of the system is
> independent of the g one chooses. Since g is public, and since the security
> of the system does not suffer for just choosing the same g every time,
> Wideskies just says "screw it, we're going with 1+N".

I'm curious, but I'll draw the line there and trust :-)

>> How do you want to receive proposed changes to the PDF, if at all?  I'm 
>> rephrasing
>> your version a bit as I go, and quite happy to keep it separate so there is
>> a simple version for those that don't "speak algebra".
> 
> I've created another issue, PIRK-70, to add the LaTeX beamer doc from which
> I generated the pdf, along with the relevant Pirk images. These docs will
> live in the main branch (not the web branch) under contrib. This is in
> addition to the changes I've made to the PDF to clarify the math language.
> With the TeX there, anyone can edit the doc, if they so choose. Tim, you
> could do that, or roll your own in whatever format you like.

Cool, thanks.  I'll keep working my way through it as I get a few mins.

Please can you clarify, the doc for Wideskies algorithm (slide 16) says
that zeta is chosen to be in (Z/N^2 Z)* -- but the code we have in
Paillier.java[1] appears to be selecting for (Z/NZ)*

[1]
https://github.com/apache/incubator-pirk/blob/master/src/main/java/org/apache/pirk/encryption/Paillier.java#L257

Regards,
Tim

> What do you think?
> 
> Walter
> 
> On Tue, Sep 20, 2016 at 8:33 AM, Tim Ellison  wrote:
> 
>> On 19/09/16 18:36, Walter Ray-Dulany wrote:
>>> Apologies! It's disorienting at first, and most of all when one actually
>>> tries to sit down and do a real example. The version on the slides was
>> not
>>> written in one go, I assure you.
>>>
>>> Let's go through, and see what's not working.
>>
>> I appreciate your patience Walter.  I'm sure this "just makes sense" to
>> you, but my brain is hurting trying to keep up :-)
>>
>>> **
>>>
 I'm trying a very simple example.  I'm going to choose, p = 3, q = 5 and
>>> a message m = 42
>>>
>>> Already we're in trouble. p and q are fine; but remember that the
>> plaintext
>>> space (let's call it P(N)) is the set of all integers in Z/NZ; that is,
>> it
>>> is all numbers m
>>>
>>> 0 <=  m < N
>>
>> Ah, so understanding the notation was confusing me, which is not going
>> to help!  Is this the same as I've seen this written elsewhere as double
>> stroked Z subscript N?
>>
>>> You can see already that the m you chose is not in the plaintext space.
>>>
>>> Let's pick a new m to continue with; in this case, let's choose your m,
>> but
>>> mod 15 so that it lies in P(N). Thus, our new m going forward shall be
>>>
>>> m = 12
>>
>> Interesting.  That means that the size of message I can encode directly
>> using this scheme is related to the bit length of my RSA factors.
>>
>> I assume that, practically, it is better to work with smaller factors to
>> make the math operations faster (but not too small to compromise
>> security) and chunk the message accordingly, rather than working with
>> large messages.
>>
>>> **
>>>
 I'm going to pick g = 240.  I think it needs to be a multiple of N that
>>> is greater than N*N, correct?
>>>
>>> No, and this is important. g has to be an element of (Z/(N squared )Z)*
>> of
>>> order a nonzero multiple of N. That sentence is meaningless unless you're
>>> already embedded in the mathematics, so let's go through what it means,
>> bit
>>> by bit.
>>>
>>> g must be:
>>> 1. *an element of (Z/(N squared)Z)**: everything but the outer * on the
>>> right just means that 0 <= g < N*N; in this case that means 0 <= g < 225.
>>> The outer * on the right indicates that we only want to take a certain
>>> special kind of g: one that is what we call a *unit* mod N*N; that is, it
>>> means that

[GitHub] incubator-pirk pull request #95: [PIRK-69] Improve clarity of group theory p...

2016-09-20 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-pirk/pull/95


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-pirk pull request #97: [PIRK-64]: Added new Cloud Instructions pag...

2016-09-20 Thread Tim Ellison 
Thanks Jacob!  I'll be trying these out shortly.

Regards,
Tim

On 20/09/16 15:29, jacobwil wrote:
> GitHub user jacobwil opened a pull request:
> 
> https://github.com/apache/incubator-pirk/pull/97
> 
> [PIRK-64]: Added new Cloud Instructions page with details on how to use…
> 
> … AWS, GCP, and (in the future) Azure with Pirk
> 
> I've made instructions for the website on how to use GCP, AWS, and 
> (hopefully, eventually) Azure to run Pirk. 
> 
> You can merge this pull request into a Git repository by running:
> 
> $ git pull https://github.com/jacobwil/incubator-pirk gh-pages
> 
> Alternatively you can review and apply these changes as the patch at:
> 
> https://github.com/apache/incubator-pirk/pull/97.patch
> 
> To close this pull request, make a commit to your master/trunk branch
> with (at least) the following in the commit message:
> 
> This closes #97
> 
> 
> commit e347cfb6a3575c70fc598413ca5e87bee451f738
> Author: Jacob Wilder 
> Date:   2016-09-20T04:09:10Z
> 
> PIRK-64: Added new Cloud Instructions page with details on how to use 
> AWS, GCP, and (in the future) Azure with Pirk
> 
> 
> 
> 
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

[GitHub] incubator-pirk pull request #97: [PIRK-64]: Added new Cloud Instructions pag...

2016-09-20 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-pirk/pull/97


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-pirk pull request #96: [PIRK-70] Add supporting LaTeX for math dec...

2016-09-20 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-pirk/pull/96


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-pirk pull request #97: [PIRK-64]: Added new Cloud Instructions pag...

2016-09-20 Thread jacobwil 
GitHub user jacobwil opened a pull request:

https://github.com/apache/incubator-pirk/pull/97

[PIRK-64]: Added new Cloud Instructions page with details on how to use…

… AWS, GCP, and (in the future) Azure with Pirk

I've made instructions for the website on how to use GCP, AWS, and 
(hopefully, eventually) Azure to run Pirk. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jacobwil/incubator-pirk gh-pages

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-pirk/pull/97.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #97


commit e347cfb6a3575c70fc598413ca5e87bee451f738
Author: Jacob Wilder 
Date:   2016-09-20T04:09:10Z

PIRK-64: Added new Cloud Instructions page with details on how to use AWS, 
GCP, and (in the future) Azure with Pirk




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-pirk pull request #96: [PIRK-70] Add supporting LaTeX for math dec...

2016-09-20 Thread wraydulany 
GitHub user wraydulany opened a pull request:

https://github.com/apache/incubator-pirk/pull/96

[PIRK-70] Add supporting LaTeX for math deck to repo.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wraydulany/incubator-pirk PIRK-70

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-pirk/pull/96.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #96


commit 9186a3d769940d3c024fa3f74ea3a56a49118f25
Author: Walter Ray-Dulany 
Date:   2016-09-20T14:14:40Z

Added files.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

2016-09-20 Thread Tim Ellison 
On 19/09/16 22:10, Suneel Marthi wrote:
> Here's an example from the Flink project for how they go about new features
> or system breaking API changes, we could start a similar process. The Flink
> guys call these FLIP (Flink Improvement Proposal) and Kafka community
> similarly has something called KLIP.
> 
> We could start a PLIP (??? :-) )

I expect an change 'process' will evolve as required, but +1 for
starting to document some of the architecture on the Pirk wiki.

My notebook has a few scrappy line drawings at the moment.

Regards,
Tim


> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
> 
> 
> On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi 
> wrote:
> 
>> A shared Google doc would be more convenient than a bunch of Jiras. Its
>> easier to comment and add notes that way.
>>
>>
>> On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson 
>> wrote:
>>
>>> Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
>>> Based off my pirk-63 I was able to pull spark and storm out with no
>>> issues.  I was planning to pull them out, then tackling elastic search,
>>> then hadoop as it's a little entrenched.  This should keep most PRs to
>>> manageable chunks. I think once that's done addressing the configs will
>>> make more sense.
>>>
>>> I'm open to suggestions. But the hope would be:
>>> Pirk-parent
>>> Pirk-core
>>> Pirk-hadoop
>>> Pirk-storm
>>> Pirk-parent
>>>
>>> Pirk-es is a little weird as it's really just an inputformat, seems like
>>> there's a more general solution here than creating submodules for every
>>> inputformat.
>>>
>>> Darin
>>>
>>> On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
>>>

>>>
 Refactor is definitely a first priority.  Is there a design/proposal
>>> draft
 that we could comment on about how to go about refactoring the code.  I
 have been trying to keep up with the emails but definitely would have
 missed some.



 On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
 eawilli...@apache.org > wrote:

> Agree - let's leave the config/CLI the way it is for now and tackle
>>> that as
> a subsequent design discussion and PR.
>
> Also, I think that we should leave the ResponderDriver and the
> ResponderProps alone for this PR and push to a subsequent PR (once we
> decide if and how we would like to delegate each).
>
> I vote to remove the 'platform' option and the backwards compatibility
>>> in
> this PR and proceed with having a ResponderLauncher interface and
>>> forcing
> its implementation by the ResponderDriver.
>
> And, I am not so concerned with having one fat jar vs. multiple jars
>>> right
> now - to me, at this point, it's a 'nice to have' and not a 'must
>>> have'
>>> for
> Pirk functionality. We do need to break out Pirk into more clearly
>>> defined
> submodules (which is in progress) - via this re-factor, I think that
>>> we
> will gain some ability to generate multiple jars which is nice.
>
>
>
> On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison 
> wrote:
>
>> On 19/09/16 15:46, Darin Johnson wrote:
>>> Hey guys,
>>>
>>> Thanks for looking at the PR, I apologize if it offended anyone's
> eyes:).
>>>
>>> I'm glad it generated some discussion about the configuration.  I
> didn't
>>> really like where things were heading with the config.  However,
>>> didn't
>>> want to create to much scope creep.
>>>
>>> I think any hierarchical config (TypeSafe or yaml) would make
>>> things
> much
>>> more maintainable, the plugin could simply grab the appropriate
>>> part of
>> the
>>> config and handle accordingly.  I'd also cut down the number of
>>> command
>>> line options to only those that change between runs often (like
>>> input/output)
>>>
 One option is to make Pirk pluggable, so that a Pirk installation
> could
 use one or more of these in an extensible fashion by adding JAR
>>> files.
 That would still require selecting one by command-line argument.
>>>
>>> An argument for this approach is for lambda architecture
>>> approaches
> (say
>>> spark/spark-streaming) were the contents of the jars would be so
> similar
>> it
>>> seems like to much trouble to create separate jars.
>>>
>>> Happy to continue working on this given some direction on where
>>> you'd
>> like
>>> it to go.  Also, it's a bit of a blocker to refactoring the build
>>> into
>>> submodules.
>>
>> FWIW my 2c is to not try and fix all the problems in one go, and
>>> rather
>> take a compromise on the configurations while you tease apart the
>> submodules in to separate source code trees, poms, etc; then come
>>> back
>> and fix the runtime configs.
>>
>>

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

2016-09-20 Thread Tim Ellison 
On 19/09/16 21:22, Darin Johnson wrote:
> Alright, that was in the spirit of what I was thinking when I did this.
> 
> Why don't we take Tim's suggested improvements to my PR (I'll do the
> necessary cleanup) and at the same time just remove the platform argument
> altogether since backwards compatibility isn't upsetting anyone.
> 
> We'll still need a command line option for the launcher for now as we don't
> have submodules we can decide between the two choices after we break out
> submodules and improve the config.

Sounds great - let me know how I can help.

Regards,
Tim


> On Sep 19, 2016 12:19 PM, "Tim Ellison"  wrote:
> 
>> On 19/09/16 15:46, Darin Johnson wrote:
>>> Hey guys,
>>>
>>> Thanks for looking at the PR, I apologize if it offended anyone's eyes:).
>>>
>>> I'm glad it generated some discussion about the configuration.  I didn't
>>> really like where things were heading with the config.  However, didn't
>>> want to create to much scope creep.
>>>
>>> I think any hierarchical config (TypeSafe or yaml) would make things much
>>> more maintainable, the plugin could simply grab the appropriate part of
>> the
>>> config and handle accordingly.  I'd also cut down the number of command
>>> line options to only those that change between runs often (like
>>> input/output)
>>>
 One option is to make Pirk pluggable, so that a Pirk installation could
 use one or more of these in an extensible fashion by adding JAR files.
 That would still require selecting one by command-line argument.
>>>
>>> An argument for this approach is for lambda architecture approaches (say
>>> spark/spark-streaming) were the contents of the jars would be so similar
>> it
>>> seems like to much trouble to create separate jars.
>>>
>>> Happy to continue working on this given some direction on where you'd
>> like
>>> it to go.  Also, it's a bit of a blocker to refactoring the build into
>>> submodules.
>>
>> FWIW my 2c is to not try and fix all the problems in one go, and rather
>> take a compromise on the configurations while you tease apart the
>> submodules in to separate source code trees, poms, etc; then come back
>> and fix the runtime configs.
>>
>> Once the submodules are in place it will open up more work for release
>> engineering and tinkering that can be done in parallel with the config
>> polishing.
>>
>> Just a thought.
>> Tim
>>
>>
>>> On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison 
>> wrote:
>>>
 On 19/09/16 13:40, Ellison Anne Williams wrote:
> It seems that it's the same idea as the ResponderLauncher with the
 service
> component added to maintain something akin to the 'platform'. I would
> prefer that we just did away with the platform notion altogether and
>> make
> the ResponderDriver 'dumb'. We get around needing a platform-aware
 service
> by requiring the ResponderLauncher implementation to be passed as a CLI
 to
> the ResponderDriver.

 Let me check I understand what you are saying here.

 At the moment, there is a monolithic Pirk that hard codes how to respond
 using lots of different backends (mapreduce, spark, sparkstreaming,
 storm , standalone), and that is selected by command-line argument.

 One option is to make Pirk pluggable, so that a Pirk installation could
 use one or more of these in an extensible fashion by adding JAR files.
 That would still require selecting one by command-line argument.

 A second option is to simply pass in the required backend JAR to select
 the particular implementation you choose, as a specific Pirk
 installation doesn't need to use multiple backends simultaneously.

 ...and you are leaning towards the second option.  Do I have that
>> correct?

 Regards,
 Tim

> Am I missing something? Is there a good reason to provide a service by
> which platforms are registered? I'm open...
>
> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison 
 wrote:
>
>> How about an approach like this?
>>https://github.com/tellison/incubator-pirk/tree/pirk-63
>>
>> The "on-ramp" is the driver [1], which calls upon the service to find
>> a
>> plug-in [2] that claims to implement the required platform responder,
>> e.g. [3].
>>
>> The list of plug-ins is given in the provider's JAR file, so the ones
>> we
>> provide in Pirk are listed together [4], but if you split these into
>> modules, or somebody brings their own JAR alongside, these would be
>> listed in each JAR's services/ directory.
>>
>> [1]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/java/org/apache/pirk/responder/wideskies/
>> ResponderDriver.java
>> [2]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
>> [3]
>>