Re: building on OSX 10.12.1

[J Patel]

I think the right way to do this is to move to the latest version of their
libraries. Things like syscall are now deprecated in OSX.

I think Zuyu and Harshad may have some insights on how to do this right.

On Tue, Nov 22, 2016 at 8:49 PM Marc Spehlmann 
wrote:

> That link appears to be broken. Do you mind copying the text?
>
> On Tue, Nov 22, 2016 at 8:24 PM, Navneet Sankara  wrote:
>
> > Probably this macOS Sierra issue:
> > https://lists.apache.org/thread.html/81e200d3599f519009ebe54674f4aa
> > 96c71f3e22ccbacaf9e5198332@
> > 
> >
> > On Tue, Nov 22, 2016, 19:50 Marc Spehlmann 
> wrote:
> >
> > > Hello quickstep,
> > > I'm getting compile errors with the third party libraries. I have some
> of
> > > these installed on my system (protobuf). Is there a flag to tell CMake
> to
> > > use the system library instead of building the bundled one?
> > >
> >
>

Re: Accidental Push to Master

[J Patel]

I don't think so :-( ASF, I think, is plain old Git.

On Friday, September 23, 2016, Navneet Potti  wrote:

> Github has a new feature called protected branches <
> https://help.github.com/articles/about-protected-branches/> that
> automatically requires a PR acceptance before code can be merged into a
> branch. Is there a similar functionality for ASF repo?
>
>
>
> > On Sep 23, 2016, at 15:49, Hakan Memisoglu  > wrote:
> >
> > I accidentally pushed my changes to master directly, then I revert the
> changes with forced push. In original ASF repo, everything seems alright.
> In Github, the changes have not been reflected yet.
> >
> > Again sorry for the inconvenience.
> >
>
>

List of potential work to do on Quickstep

[J Patel]

Hi folks,

Here is a list of features that would be good for the community to work on.
Feel free to add or comment on this list.

1: Improve handling of aggregation: Aggregate handling in Quickstep is slow
as a separate hash table is being built for each aggregate. PR
https://github.com/apache/incubator-quickstep/pull/90 is a step in fixing
this, but there is more to be done, including increasing the space
efficiency of the hash table, improving the finalize operation (which is
single-threaded), and considering partitioning (so that finalize can be
parallelized).

2: The use of ColumnVectors is very expensive as it involves a full extra
read and write of data, and results in a bad memory access pattern. That
design needs to be rethought/refactored. Nav has suggested using an
iterator model v/s accessors and that is a good idea. We can probably go
beyond that and think of defining patterns for taking an input, applying a
predicate, and applying a projection (copy). Any ideas here are welcome.

3: We have bloomfilters and that needs to be optimized to work with joins.
Jianqiao is working on this.

4: Error handling in the system can be improved. Here we need to consider
if we want to use error return codes or C++ throw/catch mechanism. Right
now we use a mix of both. I am starting to turn in favor of throw/catch as
that way we at least have a way of catching the error at the top (rather
than crashing). We can then refactor the code to add entire throw/catch
chains. Right now the most serious error handling that is lacking, IMHO, is
when we are loading a large file and there is a corrupted tuple near the
end. The system crashes after making the user wait, and there is no
cleanup.

5: Our type system also needs a major surgery to make it easier to add new
types. Clean UDFs support is also missing.

Other thoughts?

Cheers,
Jignesh

Re: Mentor concerns

[J Patel]

I'm on vacation - can someone take care of the website? Else, I'll do it on
Thursday.

I believe we have Jira tickets for all the main features. I'll check when
I'm back. Agreed would be good to have Jira tickets for visibility to the
community.

On a related note to perhaps clear some misconception: Some of the changes
come around when someone sees a problem and is inspired to fix it. I think
JQs plan visualization is a good example - he did it over a few days on his
own. I love that feature (I think we all do!), and discovered it when the
PR showed up. He had a great writeup in his PR (IMHO a good model for all
of us). Agreed if that didn't have a Jira ticket would be good to get in
the habit of opening one when inspiration strikes for an idea, and close it
with the final commit. But please do keep such inspiration driven features
going! I think no one wants to stop doing that.

And I very much would love for existing folks to take the lead like JQ did!


On Sunday, August 14, 2016, Roman Shaposhnik  wrote:

> Hi!
>
> Quickstep community is coming along fine, but,
> with my mentor hat on, I still have a few concerns:
>   1. The Download button needs to be removed from
>   the website until there's a first official release
>
>   2. Website has to have an Incubator disclaimer:
>  http://incubator.apache.org/guides/branding.html#disclaimers
>
>3. I see tons of GitHub commits and yet I see pretty much
>no dev@ list or JIRA traffic. Why is this code being commited?
>Where is it discussed? This all is a mystery to me and I suspect
>anybody else who's not on-site with those who commit.
>
> I'd like to ask that the community quickly takes care of #1 and #2
> and comes up with an explanation for #3
>
> Thanks,
> Roman.
>

Re: 5 things

[J Patel]

Dear Julian,

Your feedback is very welcome and valuable as usual. Comments inlined
below, but overall I thought I was proposing that seems to be in line with
the Apache governance model. If there are specific issues, would love if
you can point it out in general and I can go work on it.

> In general, it is expected that to maintain
> > membership in the Committer group, the developer must be active in the
> > project in the preceding 6-month period.
>
> Procedures to allow committers and PMC members to “go emeritus” do
> exist[1] but in my opinion they are solving a non-problem.
>

Great. Sounds like this part is ok then.


> > PMC members can commit code directly to the main repository, but are
> > generally advised to open a PR and allow another Committer to close the
> PR.
>
> The role of the PMC (actually PPMC while Quickstep is incubating) is
> solely governance. PPMC members should not have more rights than committers
> where it comes to code. We don’t want to create two tiers of committers.
>

Good point -- will remove this distinction.


>
> > A Contributor can become a Committer by getting support from at least two
> > existing Committers. It is expected that a Contributor will have
> > demonstrated proficiency in understanding and working with the core
> engine
> > to become part of the Committer group.
>
> Per Apache rules committers are appointed by achieving consensus
> (typically a vote) in the PMC[2].
>

Got it. I read [2] as being general guidelines but not binding. It is okay
to do this by consensus too over email.


> > The PMC meets at least annually. Meetings are announced at least two
> weeks
> > in advance on the project developer list. Any Committer is welcome to
> > propose agenda items for the meeting. The PMC chair consults the overall
> > PMC to finalize the agenda. The meeting agenda must be finalized two days
> > before the meeting and communicated on the project developer list. Agenda
> > items can include, but is not limited to, voting for changes to the PMC,
> > changes to the Committers group, and changes to the community governance
> > document. Only existing PMC members can vote on motions. For a motion to
> > carry, a majority of the votes that are cast must be in favor of the
> > motion. Ties are broken by the PMC Chair. Proxies are permitted, and must
> > be communicated to the PMC Chair prior to the meeting.
>
> Per Apache, the PMC meets continually… via email. I suppose the PMC could
> have an annual meeting, but business such as appointing members does not
> have to wait for that.
>

Agreed. I worded this carefully to say "at least" ; limit on the
upper-bound. So, I'm not seeing a conflict there.


> Apache has a procedure for appointing members to the PMC[3]. You need to
> follow this.
>

I did get this part wrong. Nice catch! I should really propose simply
following the guidelines there. I was hoping there was some way for the
existing PPMC to agree on a proposed new PPMC member before emailing
bo...@apache.org. So, perhaps what I have could be applied to getting
consensus on a proposal within the Quickstep PPMC and then making a formal
proposal.


> Jignesh, As I said in a previous email, it’s good that you are thinking
> about governance. But we are going to be at odds until you read &
> understand the existing Apache governance. I think you need to take the
> time to do this before proposing alternatives. The Apache governance is
> well established and is proven. And considerable parts of it are mandatory.
>

If the above largely covers it, I'll work on making another round over the
next few days.

Agreed Julian  about your point about me not fully understanding the
governance. It is somewhat complicated, and in some cases information is a
bit scattered. I am still learning about the Apache way, and very much
appreciate your detailed input.

Cheers,
Jignesh

Re: [jira] [Commented] (QUICKSTEP-38) Enhance python scripts to support Python3

[J Patel]

Good point on adding changes to our git workflow.

@Zuyu - I see you are on the wiki in edit mode, so will let you make the
changes. Thanks in advance!

On Wed, Aug 3, 2016 at 1:58 PM, ASF GitHub Bot (JIRA) 
wrote:

>
> [
> https://issues.apache.org/jira/browse/QUICKSTEP-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406422#comment-15406422
> ]
>
> ASF GitHub Bot commented on QUICKSTEP-38:
> -
>
> Github user cwelton commented on the issue:
>
> https://github.com/apache/incubator-quickstep/pull/75
>
> Thanks for fixing, I suggest adding a contributor's guide with your
> suggested git workflow, to help streamline future
> contributors/contributions.
>
>
> > Enhance python scripts to support Python3
> > -
> >
> > Key: QUICKSTEP-38
> > URL: https://issues.apache.org/jira/browse/QUICKSTEP-38
> > Project: Apache Quickstep
> >  Issue Type: Improvement
> >  Components: Utility
> >Reporter: Caleb Welton
> >Assignee: Caleb Welton
> >Priority: Minor
> >
> > Currently cyclic_dependency.py and validate_cmakelists.py both only
> support python2.  I propose updating them to support both python2 and
> python3.
> > Reason, Python 3 is has reached a level of maturity that it can be
> considered ready for prime time, adding support for Python 3 is a minimal
> cost and increases both future readiness and availability on more systems.
> The change to support both versions of the language is relatively small and
> non-invasive.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Re: Podling Report Reminder - August 2016

[J Patel]

Thanks Julian for the reminder! Just added the report for Quickstep.

Cheers,
Jignesh

On Wed, Aug 3, 2016 at 2:57 PM, Julian Hyde  wrote:

> Today is the deadline for the report. If you miss the deadline, this will
> be the second consecutive month that you have failed to report. The IPMC
> and Board do not appreciate multiple missed reports.
>
> Julian
>
>
> > On Jul 27, 2016, at 10:18 AM, Julian Hyde  wrote:
> >
> > Quickstep missed its report last month, so it is very important to file
> this month.
> >
> > Who is going to write the report?
> >
> > Julian
> >
> >
> >> On Jul 27, 2016, at 4:47 AM, johndam...@apache.org wrote:
> >>
> >> Dear podling,
> >>
> >> This email was sent by an automated system on behalf of the Apache
> >> Incubator PMC. It is an initial reminder to give you plenty of time to
> >> prepare your quarterly board report.
> >>
> >> The board meeting is scheduled for Wed, 17 August 2016, 10:30 am PDT.
> >> The report for your podling will form a part of the Incubator PMC
> >> report. The Incubator PMC requires your report to be submitted 2 weeks
> >> before the board meeting, to allow sufficient time for review and
> >> submission (Wed, August 03).
> >>
> >> Please submit your report with sufficient time to allow the Incubator
> >> PMC, and subsequently board members to review and digest. Again, the
> >> very latest you should submit your report is 2 weeks prior to the board
> >> meeting.
> >>
> >> Thanks,
> >>
> >> The Apache Incubator PMC
> >>
> >> Submitting your Report
> >>
> >> --
> >>
> >> Your report should contain the following:
> >>
> >> *   Your project name
> >> *   A brief description of your project, which assumes no knowledge of
> >>   the project or necessarily of its field
> >> *   A list of the three most important issues to address in the move
> >>   towards graduation.
> >> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> >>   aware of
> >> *   How has the community developed since the last report
> >> *   How has the project developed since the last report.
> >>
> >> This should be appended to the Incubator Wiki page at:
> >>
> >> http://wiki.apache.org/incubator/August2016
> >>
> >> Note: This is manually populated. You may need to wait a little before
> >> this page is created from a template.
> >>
> >> Mentors
> >> ---
> >>
> >> Mentors should review reports for their project(s) and sign them off on
> >> the Incubator wiki page. Signing off reports shows that you are
> >> following the project - projects that are not signed may raise alarms
> >> for the Incubator PMC.
> >>
> >> Incubator PMC
> >
>
>

Re: 5 things

[J Patel]

Thanks Julian! Nope hadn't seen those emails -- I bet its in my (now
deleted) Pivotal mailbox.

@Julian: Nice feedback on using the Apache terms. Makes sense. New proposal
attached.

@Roman: We do have a new README.md file in the main repo to get started.

@Julian and @Roman:  Feedback well taken on the release.

All: Would love to hear your thoughts on a release. I'd like to propose mid
to late November, which likely works out best given the semester cadence
(yeah -- I know we are far skewed in the Committer to that, at least for
now).


QUICKSTEP DEVELOPER COMMUNITY GOVERNANCE


Quickstep is a data platform that is written in C++ and uses advanced C++
techniques including template meta-programming and various abstractions to
balance high-performance with extensibility. Thus, a good grasp over C++
concepts is essential for a Quickstep developer.



Quickstep has a group of *Committers* who are collectively responsible for
the code repository. Committers have write access to the code repository
and must have signed a Contributors License Agreement (CLA). In addition,
there is a set of *Contributors* who contribute individual changes. The
overall stewardship of the project rests with the Project Management
Committee (*PMC*), which is a subset of the Committers. A member of the PMC
is selected as the *PMC Chair*.


Contributors and Committers must submit pull requests (PR) for proposed
code changes. Only Committers can close PRs. But, a committer can not close
their own PR, which must be examined and closed by another Committer.
Anyone can open a PR and once a PR has been accepted that developer becomes
part of the Contributors group. In general, it is expected that to maintain
membership in the Committer group, the developer must be active in the
project in the preceding 6-month period.


PMC members can commit code directly to the main repository, but are
generally advised to open a PR and allow another Committer to close the PR.


A Contributor can become a Committer by getting support from at least two
existing Committers. It is expected that a Contributor will have
demonstrated proficiency in understanding and working with the core engine
to become part of the Committer group.


The PMC meets at least annually. Meetings are announced at least two weeks
in advance on the project developer list. Any Committer is welcome to
propose agenda items for the meeting. The PMC chair consults the overall
PMC to finalize the agenda. The meeting agenda must be finalized two days
before the meeting and communicated on the project developer list. Agenda
items can include, but is not limited to, voting for changes to the PMC,
changes to the Committers group, and changes to the community governance
document. Only existing PMC members can vote on motions. For a motion to
carry, a majority of the votes that are cast must be in favor of the
motion. Ties are broken by the PMC Chair. Proxies are permitted, and must
be communicated to the PMC Chair prior to the meeting.


*Core Contributors, also the PMC (as of August 5, 2016):*
Harshad Deshmukh
Rogers Jeffrey Leo John
Hakan Memisoglu
Jignesh M. Patel (PMC Chair)
Navneet Potti
Saket Saurabh
Marc Spehlmann
Zuyu Zhang

Jianqiao Zhu

*Core Collaborators (as of August 5, 2016):*
Shoban Chandrabose
Craig Chasseur
Shixuan Fan
Julian Hyde
Yinan Li
Rathijit Sen
Roman Shaposhnik
Adalbert Gerald Soosai Raj
Siddharth Suresh
Caleb Welton

> Qiang Zeng

Re: Greetings

[J Patel]

+1 from me.

Zuyu is our resident expert on this, so would love to hear his input too.

Thanks Caleb!

Cheers,
Jignesh

On Monday, August 1, 2016, Caleb Welton  wrote:

> Hello Quickstep team!
>
>
> A. This list is a bit difficult to find.
>
> I'd like to suggest adding a link to it from
> https://cwiki.apache.org/confluence/display/QUICKSTEP/Quickstep+Home or
> http://quickstep.incubator.apache.org/
>
>
> B. I'd like to propose two minor enhancements.
>
> 1. Update cyclic_dependency.py and validate_cmakelists.py to support both
> Python2 and Python3.
>
> Reason, Python 3 is has reached a level of maturity that it can be
> considered ready for prime time, adding support for Python 3 is a minimal
> cost and increases both future readiness and availability on more systems.
> The change to support both versions of the language is relatively small and
> non-invasive.
>
> If the community is interested I have a patch.
>
>
> 2. Update cyclic_dependency.py and validate_cmakelists.py to move
> configuration to a separate configuration file.
>
> Reason, currently both scripts have intermingled code and configuration
> which means that any new ignored dependencies require changes to the code
> rather than configuration.  The scripts seem generally useful and I would
> like to incorporate them into a different project, it would be nice to be
> able to do so without requiring code changes so that the code can be used
> as is without modification.
>
> I am willing to develop a patch if the community is interested.
>
>
> Cheers!
>Caleb
>