[DISCUSS] Need for standard driver?

[Marc Spehlmann]

I have a PR out for a CLI which interfaces with the network over gRPC [1].
Julian recommended using a package called Avantica which wraps over a
standard JDBC+ODBC drivers and takes care the protocol.

The question I have for /dev is if we have a need right now for a standard
driver. I could see a use case being that for demoing QS and trying to find
users, it would be valuable.

Does anyone have comments about how to do interaction with QS?

--Marc

[1] https://issues.apache.org/jira/browse/QUICKSTEP-87

Re: Blog post on storage formats

[Marc Spehlmann]

Cool, thanks Harshad! Looks good.

One day we will come up with better  css but sparse simplicity
trumps elaborate clutter so this is good.

On Thu, Apr 6, 2017 at 7:47 AM, Harshad Deshmukh 
wrote:

> Hello,
>
> Thanks to Julian, Jignesh and Marc for your suggestions! The post is up
> now - http://quickstep.incubator.apache.org/guides/2017/03/30/stor
> age-formats-quickstep.html
>
>
>
> On 03/31/2017 06:18 PM, Julian Hyde wrote:
>
>> Very nice! And pictures too!
>>
>> One suggestion (take it or leave it) - use colors to indicate which
>> values belong to which columns. If will make the difference between row-
>> and column-oriented format more apparent.
>>
>> At some point those 3 graphics, plus the table with the data set, could
>> be boiled down to one slide for presentations. Similar to the one that
>> Arrow use[1].
>>   Julian
>>
>>
>> [1] https://arrow.apache.org/img/simd.png > simd.png>
>>
>> On Mar 31, 2017, at 3:33 PM, Jignesh Patel 
>>> wrote:
>>>
>>> Very nice Harshad! I made minor edits in the PR
>>> https://github.com/hbdeshmukh/incubator-quickstep-site/pull/1
>>>
>>> Cheers,
>>> Jignesh
>>>
>>> On 3/31/17, 2:29 PM, "Harshad Deshmukh"  wrote:
>>>
>>> Hello all,
>>>
>>> I have written a blog post on various storage formats implemented in
>>> Quickstep. Please review the pull request -
>>> https://github.com/apache/incubator-quickstep-site/pull/2 and
>>> provide
>>> your feedback. If you don't have jekyll set up on your machine, you
>>> can
>>> use this link to read the post:
>>> https://github.com/hbdeshmukh/incubator-quickstep-site/blob/
>>> storage-formats-post/_posts/2017-03-30-storage-formats-quick
>>> step.markdown
>>>
>>> The SQL formatting doesn't look well on GitHub, but I think it shows
>>> up
>>> alright on the actual website.
>>>
>>> --
>>>
>>> Thanks,
>>> Harshad
>>>
>>>
>>>
>>>
>>>
>>
> --
> Thanks,
> Harshad
>
>

Re: quickstep server

[Marc Spehlmann]

I think that in the initial version, it will not be aware of multiple
clients, but in the second version, this will not be hard to accommodate.

On Sat, Apr 1, 2017 at 4:48 PM, Zuyu Zhang  wrote:

> Hi Marc,
>
> With the NetworkWrapper, does the QuickstepServer could serve queries from
> multiple clients?
> ​
> Cheers,
> Zuyu
>

Re: quickstep server

[Marc Spehlmann]

Hi Zuyu,
Yes, that helps. I am making a prototype right now where I took the
QuickstepCli code and made another executable, QuickstepServer. The server
does all of the same initialization process. The main difference is that,
instead of communicating with a linereader, it must communicate with a
NetworkWrapper. Also, communication must not be one-way and rely on prints
to the screen, but it is two way and sends the results of STDOUT and ERR to
the NetworkWrapper.

Once the prototype is complete, we'll have a better idea if the CLI can be
refactored to save some code duplication.

--Marc

On Fri, Mar 31, 2017 at 3:06 PM, Zuyu Zhang  wrote:

> Hi Marc,
>
> The distributed version uses the network-based implementation of TMB to
> communicate among multiple components, including DistributedCli, Conductor
> (which contains Foreman), and Executor (which contains a Worker pool). We
> have defined additional TMB message types for the communication.
>
> For your purpose, you might want to split QuickstepCli.cpp into two new
> executables, one for the cli, and one the database core. Both need to link
> the Protobuf 3 and gRPC. And then, the python program could talk to the cli
> executable to send a query or a command.
>
> Hope it helps.
>
> Cheers,
> Zuyu
>

Re: quickstep server

[Marc Spehlmann]

Thank you. I spoke with Tianrun yesterday. Harshad brought up the fact that
Zuyu might also have made changes in the distributed version which can be
ported back for our use in the single node world.

@Zuyu, is there distributed code that would be good to borrow instead of
rolling our own version?

On Thu, Mar 30, 2017 at 12:58 PM, Jianqiao <jianq...@cs.wisc.edu> wrote:

> Hi Marc,
>
> Tianrun had one implementation of the TCP Server interface when he was
> working on the GRAIL related stuff. The idea is to subclass the LineReader
> interface. This minimizes the changes needed inside QuickstepCli.cpp. The
> code might help and is available here:
>
> https://bitbucket.org/TianrunLi/tcpreader/src
>
> Best,
> Jianqiao
>
> 2017-03-30 10:10 GMT-05:00 Marc Spehlmann <sp...@apache.org>:
>
> > Hi dev,
> > I'm looking to make a simple TCP server interface to Quickstep so that we
> > can keep the db running while we communicate to it from a python program.
> > After investigating, I think the simplest way to do this is:
> >
> > - create a Server executable which:
> >   - opens a TCP socket
> >   - forks a child with an open pipe
> > - std input of the child process is redirected from the pipe
> > - child process launches the main() code of QuickstepCli and goes on
> as
> > normal
> >   - Server process channels messages from the TCP buffer to the pipe
> >   - Server exits when child exits
> >
> > Since I'd like to reuse the code in main() of quickstep_cli, I think this
> > means abstracting that code into a stand alone library.
> >
> > Does anyone have any initial comments on this design?
> >
> > --Marc
> >
>

quickstep server

[Marc Spehlmann]

Hi dev,
I'm looking to make a simple TCP server interface to Quickstep so that we
can keep the db running while we communicate to it from a python program.
After investigating, I think the simplest way to do this is:

- create a Server executable which:
  - opens a TCP socket
  - forks a child with an open pipe
- std input of the child process is redirected from the pipe
- child process launches the main() code of QuickstepCli and goes on as
normal
  - Server process channels messages from the TCP buffer to the pipe
  - Server exits when child exits

Since I'd like to reuse the code in main() of quickstep_cli, I think this
means abstracting that code into a stand alone library.

Does anyone have any initial comments on this design?

--Marc

0.1.0 release notes

[Marc Spehlmann]

Since this is the first release and there are a lot of features of
Quickstep which we have our specialities in, we might want to group edit
our release notes page. In the future this will be easier as it will be a
few months of changes rather than several years.

Here is the link
https://cwiki.apache.org/confluence/display/QUICKSTEP/0.1.0

And the release notes landing page:
https://cwiki.apache.org/confluence/display/QUICKSTEP/Release+Notes

--Marc

Re: release announcement

[Marc Spehlmann]

Thank you Julian. I took a look at the archives [1] to get an idea of what
is the norm. I also made note of the things you mentioned, plus a few more
tidying up issues.

I'm holding off on the announce email until I get a few other people's
input, and will send it by the end of today

--Marc

[1] https://www.mail-archive.com/announce@apache.org/

On Sat, Mar 25, 2017 at 11:27 AM, Julian Hyde <jh...@apache.org> wrote:

> Send it to the announce list. You can see other announcements and get some
> ideas here: http://mail-archives.apache.org/mod_mbox/www-announce/
> 201703.mbox/browser <http://mail-archives.apache.
> org/mod_mbox/www-announce/201703.mbox/browser>
>
> Yes, it’s an opportunity to describe your mission (and unique technology)
> in a bit more detail. The list has a wide audience (e.g. I get pings from
> analysts after announcing a calcite release there) so use the exposure
> widely.
>
> Also, it can be the first tweet on that long-neglected
> https://twitter.com/ApacheQuickstep <https://twitter.com/ApacheQuickstep>
> account.
>
> On the downloads page I would include the date of the release and a link
> to the release notes/history. You will want to describe the baseline
> functionality that is in this release so that next release you can describe
> what is new. At this point some projects have a “status” page describing in
> bullet points what is working and what is not working.
>
> Julian
>
> > On Mar 25, 2017, at 8:07 AM, Harshad Deshmukh <hars...@cs.wisc.edu>
> wrote:
> >
> > Hi Marc,
> >
> > I don't know about the Apache protocol, so may be the mentors may have
> something to say about that.
> >
> > How about saying high performance,  relational data processing engine
> developed in C++?
> >
> > The release is very exciting! I will start another thread regarding
> publicity of the release.
> >
> > Sent from Outlook for Android
> >
> >
> > From: Marc Spehlmann
> > Sent: Saturday, March 25, 9:57 AM
> > Subject: release announcement
> > To: dev@quickstep.incubator.apache.org
> >
> > Hello quickstep, I am going to send an announce email to a few of the
> other apache lists later today. Is there anything special about this
> release that we want to mention (besides that it's our first release)? If
> not, that's fine, I will send this email: ___ The Apache Quickstep
> (incubating) team is pleased to announce the release of Quickstep 0.1.0
> (our very first release!) Quickstep is a high-performance C++ SQL database.
> The release is available at: https://quickstep.incubator.
> apache.org/release Thank you, The Apache Quickstep (incubating) team ___
> --Marc
>
>

Re: release announcement + thinking about demos

[Marc Spehlmann]

I think the terse description is good. Though if we want to use
'serverless', I think it might be good to have a demo or a blog post
describing how it's used that way. (maybe we do already -- I know Hakan was
using openLambda at one point w/ qs).

__

The demo idea sounds cool. Quickstep can run benchmarks really really fast,
but I think you're right in that having a demo with a use case that would
convince users  of QS's other uses would be good for adoption. Especially
if the serverless community is lacking a relational database.



On Sat, Mar 25, 2017 at 5:02 PM, Jignesh Patel <jmp.quicks...@gmail.com>
wrote:

> Great idea Marc about making an announcement. And a big CONGRATULATIONS to
> you, the team and our mentors for getting us here. This is a huge
> accomplishment!!
>
> Building on Harshad’s idea for a terse description, how about: High
> performance auto-managed SQL data processing platform that can be used in a
> variety of settings, including serverless frameworks.
>
> IMHO, I think the self-managed/zero-knobs approach aspect of Quickstep is
> quite timely given the next evolution beyond traditional micro-services
> (which really never did data quite right). We need to emphasize it more
> centrally going forward, I think.
>
> More concretely, I’m wondering if it might make sense to put a demo
> together with Airflow or TensorFlow to highlight this aspect of Quickstep.
> We can pick a general-purpose programming language, such as Python, and
> generate a simple API to Quickstep from that language. The serverless
> nature of Quickstep means there is zero configuration needed. This is sweet
> for rapid prototyping, and even more.
>
> Assuming we have a Python API (both TensorFlow and Airflow are friendly to
> this API), we can think of building a data workflow that has some aspect of
> data ingest, cleaning, selecting/subsetting, and model building. Quickstep
> could be used in the data-heavy portions.
>
> What would be so cool about this demo? Imagine the data is stored in a
> cloud file system. Load up this workflows in a container and it works
> without any database configuration (you can’t get that with any other data
> platform that I can think of). Want to upgrade? Simply spin up a container
> with more resources, rerun the pipeline. Quickstep auto-detects,
> auto-scales, and does not require any parameter tuning. Zero knobs is ideal
> for such deployments that I think are crucial, and not just for “toy”
> settings. Today you can go get a 64core boxes with TB memory for about a
> dollar/hour (if you allow pre-emption). You can do a lot in that box, and
> Quickstep can help.
>
> I think a few demos may get more people interested in Quickstep. Comments?
> Other ideas?
>
> Cheers,
> Jignesh
>
> On 3/25/17, 10:07 AM, "Harshad Deshmukh" <hars...@cs.wisc.edu> wrote:
>
> Hi Marc,
>
> I don't know about the Apache protocol, so may be the mentors may have
> something to say about that.
>
> How about saying high performance,  relational data processing engine
> developed in C++?
>
> The release is very exciting! I will start another thread regarding
> publicity of the release.
>
> Sent from Outlook for Android
>
>
> From: Marc Spehlmann
> Sent: Saturday, March 25, 9:57 AM
> Subject: release announcement
> To: dev@quickstep.incubator.apache.org
>
> Hello quickstep, I am going to send an announce email to a few of the
> other apache lists later today. Is there anything special about this
> release that we want to mention (besides that it's our first release)? If
> not, that's fine, I will send this email: ___ The Apache Quickstep
> (incubating) team is pleased to announce the release of Quickstep 0.1.0
> (our very first release!) Quickstep is a high-performance C++ SQL database.
> The release is available at: https://quickstep.incubator.
> apache.org/release Thank you, The Apache Quickstep (incubating) team ___
> --Marc
>
>
>
>

[RESULT][VOTE] Apache Incubating Quickstep 0.1.0 RC7

[Marc Spehlmann]

Thank you to everyone who voted on the dev vote of RC7.

+1
Harshad
Josh
Jignesh
Julian (binding)

We will now submit the vote to the general incubator community.

Vote thread:
https://www.mail-archive.com/dev@quickstep.incubator.apache.org/msg02162.html

--Marc

[VOTE] Apache Incubating Quickstep 0.1.0 RC7

[Marc Spehlmann]

This vote is for approval of the Quickstep-0.1.0 RC7.

We had a few minor changes since RC6. They were changes to the LICENSE and
NOTICE
file, as well as updating some headers. These should not have made
functional changes.

Voters should download, unpack, build, and test the package before voting.

A +1 vote means that the package passes all tests and meets Apache
guidelines.
A -1 vote should be accompanied with a short reason.

Votes will remain open for 2 days in light of this being a very similar
candidate
to the last one.

___

The commit to be voted upon:
https://git-wip-us.apache.org/repos/asf?p=incubator-quickstep.git;a=shortlog;h=refs/tags/v0.1.0-rc7

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/incubator/quickstep/0.1.0/RC7/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/spehl

Please vote on releasing this package as Apache Incubating Quickstep 0.1.0.

Re: tmb compile error

[Marc Spehlmann]

adding the volatile keyword does the trick!

On Tue, Mar 7, 2017 at 3:09 PM, Zuyu Zhang  wrote:

> Btw, it seems that the compiler does not support or honor the flag
> '-Wno-unused-variable' set by the TMB.
>
> https://github.com/apache/incubator-quickstep/blob/
> master/third_party/src/tmb/CMakeLists.txt#L57
>

Re: tmb compile error

[Marc Spehlmann]

+cc zuyu wisconsin email

Zuyu this error is I believe caused by a recent change to a distributed
component that uses tmb. This is a blocker for RC6 with the HEAD of master.

If this is easy to fix, could you address it today.

The variable in question was added here:

https://github.com/apache/incubator-quickstep/commit/1cfc1c40e2bcf7ff8671d5b899a8304c9e9fd455

On Tue, Mar 7, 2017 at 1:08 PM, Jignesh Patel <jmp.quicks...@gmail.com>
wrote:

> Sadly TMB is not optional ☹
>
> On 3/6/17, 4:17 PM, "Marc Spehlmann" <spehl.apa...@gmail.com> wrote:
>
> Hi all,
> Anyone know if tmb is an optional component? I was putting together
> release
> artifacts for 0.1.0rc6 and got this compile warn->error on an ubuntu
> machine. It seems that a variable which was previously used is no
> longer
> used.
>
> tmb is a third party library. Zuyu, you mentioned it would be better to
> turn off warning->error flags for third party. What do you think would
> be
> prudent in this case?
>
> compiler:
> cxxdev@e1428abc6dae:~/incubator-quickstep/build$ cc --version
> cc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
> Copyright (C) 2015 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There
> is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> PURPOSE.
>
> error:
> [ 13%] Building CXX object
> third_party/tmb/CMakeFiles/tmb.dir/src/message_bus.cc.o
> /home/cxxdev/incubator-quickstep/release/svn-quickstep-dev/0.1.0/RC6/
> apache-quickstep-incubating-0.1.0/third_party/src/tmb/src/
> message_bus.cc:43:19:
> error: ‘tmb::tmb_receive_poll_interval_dummy’ defined but not used
> [-Werror=unused-variable]
>  static const bool tmb_receive_poll_interval_dummy =
> gflags::RegisterFlagValidator(
>^
> cc1plus: error: unrecognized command line option
> ‘-Wno-return-type-c-linkage’ [-Werror]
> cc1plus: all warnings being treated as errors
> third_party/tmb/CMakeFiles/tmb.dir/build.make:62: recipe for target
> 'third_party/tmb/CMakeFiles/tmb.dir/src/message_bus.cc.o' failed
> make[2]: *** [third_party/tmb/CMakeFiles/tmb.dir/src/message_bus.cc.o]
> Error 1
> CMakeFiles/Makefile2:1172: recipe for target
> 'third_party/tmb/CMakeFiles/tmb.dir/all' failed
> make[1]: *** [third_party/tmb/CMakeFiles/tmb.dir/all] Error 2
> Makefile:138: recipe for target 'all' failed
> make: *** [all] Error 2
>
>
>
>

Re: [VOTE] incubator-quickstep-0.1.0RC5

[Marc Spehlmann]

Thank you for the thorough feedback Julian! We will address these in the
next candidate and incorporate the Apache Rat script into our release
process.

PS. The info about how to use the KEYS is very useful- this is new
territory for all of us.

--Marc

On Mon, Mar 6, 2017 at 7:45 PM, Julian Hyde <jh...@apache.org> wrote:

> -1 (binding)
>
> Sorry my review is so late; I know that you’ve already closed the vote;
> but I figured my feedback would be more useful now than waiting for RC6.
>
> The release is actually in good shape; well done. My formal “-1” vote is
> due to issue 0 (content of the vote email) and 5 (copyright year). Please
> fix these show-stoppers in the next RC. Other issues are well worth looking
> into, because others may raise them in the IPMC vote.
>
> Julian
>
>
> 0. I found the release candidate at https://dist.apache.org/repos/
> dist/dev/incubator/quickstep/0.1.0/RC5/ <https://dist.apache.org/
> repos/dist/dev/incubator/quickstep/0.1.0/RC5/>. Please include its URL
> and its hashes (the URLs of the hashes AND the actual hash values) in the
> vote email next time. The proposition to be voted upon  needs to be
> self-contained and immutable.
>
> 1. KEYS [ https://dist.apache.org/repos/dist/dev/incubator/quickstep/KEYS
> <https://dist.apache.org/repos/dist/dev/incubator/quickstep/KEYS> ] looks
> a bit strange — usually the user & signers are at the top of each block
> (see for example http://www.apache.org/dist/calcite/KEYS <
> http://www.apache.org/dist/calcite/KEYS>) but import succeeded:
>
> $ gpg --import KEYS
> gpg: key A3C29E12: public key "Jignesh Patel <jign...@apache.org>"
> imported
> gpg: key 5A29899A: public key "Marc Spehlmann (apache) <sp...@apache.org>"
> imported
> gpg: Total number processed: 2
> gpg:   imported: 2  (RSA: 2)
>
> 2. Checked hashes and keys. (Let’s move to something stronger than MD5 and
> SHA1 soon. Also, let’s do a key signing party so that Marc & Jignesh are in
> the web of trust.)
>
> 3. The tar-ball contains KEYS. Not much point in that (because if someone
> compromised the tar-ball they would undoubtedly insert their a compromised
> KEYS file!), but no harm either.
>
> 4. I didn’t try to build (I’m not on a machine with sufficient
> horse-power), but it’s good to see clear build instructions in the tar-ball.
>
> 5. Checked that LICENSE, DISCLAIMER, NOTICE. The year in NOTICE must be
> 2017.
>
> 6. Are all of the lines in NOTICE *required*? People tend to put lines
> into a NOTICE out of *courtesy*. But it is mis-placed courtesy because the
> consumers of your software are *required* to reproduce your NOTICE
> verbatim. If Quickstep Technologies and Pivotal Software contributed under
> the ASL, then I don’t believe their copyright notices are required.
>
> In NOTICE, by “third_party/benchmark”, do you mean “third_party/src/tmb/
> benchmarks”?
>
> third_party/src/tmb/README.md says "TMB is part of the Quickstep project
> (copyright Pivotal Software, Inc.) and is distributed under the same
> license terms.” This statement is confusing now that Quickstep is now
> Apache Quickstep. If TMB has been contributed to Apache Quickstep can we
> just move it out of third_party?
>
> third_party/linenoise and third_party/gpertools don’t seem to be included
> in the tar-ball, so the entry can be removed from NOTICE.
>
> 7. It is useful (not required) to include release notes, including changes
> since the previous release. Some projects include a HISTORY.md file.
>
> 8. There are pivotalsoftware.com URLs in README.md and DEV_README.md. Can
> you change these to apache.org <http://apache.org/>.
>
> 9. Apache Rat found the following files without headers (ignoring files
> under third_party/src):
>
>   ./.gitattributes
>   ./.gitignore
>   ./.gitmodules
>   ./.travis.yml
>   ./BUILDING.md
>   ./CREDITS.md
>   ./DEV_README.md
>   ./Doxyfile
>   ./GENERATOR_FUNCTIONS.md
>   ./README.md
>   ./WORKING_WITH_AN_IDE.md
>   ./build/vagrant/README.md
>   ./parser/preprocessed/SqlLexer_gen.hpp
>   ./parser/preprocessed/genfiles.sh
>   ./query_execution/README.md
>   ./relational_operators/tests/text_scan_faulty_golden_output.txt
>   ./relational_operators/tests/text_scan_faulty_input.txt
>   ./relational_operators/tests/text_scan_golden_output.txt
>   ./relational_operators/tests/text_scan_input.txt
>   ./third_party/README.md
>   ./third_party/download_and_patch_prerequisites.sh
>   ./third_party/reset_third_party_dir.sh
>   ./third_party/patches/benchmark/benchmarkCMake.patch
>   ./third_party/patches/benchmark/benchmarkSrcCMakeLists.patch
>   ./third_party/patches/gflags/CMakeLi

tmb compile error

[Marc Spehlmann]

Hi all,
Anyone know if tmb is an optional component? I was putting together release
artifacts for 0.1.0rc6 and got this compile warn->error on an ubuntu
machine. It seems that a variable which was previously used is no longer
used.

tmb is a third party library. Zuyu, you mentioned it would be better to
turn off warning->error flags for third party. What do you think would be
prudent in this case?

compiler:
cxxdev@e1428abc6dae:~/incubator-quickstep/build$ cc --version
cc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

error:
[ 13%] Building CXX object
third_party/tmb/CMakeFiles/tmb.dir/src/message_bus.cc.o
/home/cxxdev/incubator-quickstep/release/svn-quickstep-dev/0.1.0/RC6/apache-quickstep-incubating-0.1.0/third_party/src/tmb/src/message_bus.cc:43:19:
error: ‘tmb::tmb_receive_poll_interval_dummy’ defined but not used
[-Werror=unused-variable]
 static const bool tmb_receive_poll_interval_dummy =
gflags::RegisterFlagValidator(
   ^
cc1plus: error: unrecognized command line option
‘-Wno-return-type-c-linkage’ [-Werror]
cc1plus: all warnings being treated as errors
third_party/tmb/CMakeFiles/tmb.dir/build.make:62: recipe for target
'third_party/tmb/CMakeFiles/tmb.dir/src/message_bus.cc.o' failed
make[2]: *** [third_party/tmb/CMakeFiles/tmb.dir/src/message_bus.cc.o]
Error 1
CMakeFiles/Makefile2:1172: recipe for target
'third_party/tmb/CMakeFiles/tmb.dir/all' failed
make[1]: *** [third_party/tmb/CMakeFiles/tmb.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2

Re: We really need to get going with the podling report

[Marc Spehlmann]

We have been publishing to the dev/ svn

https://dist.apache.org/repos/dist/dev/incubator/quickstep/0.1.0/RC5/

Maybe we missed something. Is there a better way to announce this?

On Thu, Mar 2, 2017 at 2:49 PM, Julian Hyde <jh...@apache.org> wrote:

> I haven’t been voting on the RCs because I haven’t yet seen a “release
> candidate” — a signed artifact (tarball) and its checksums. It seems that
> while the code is in good shape, release packaging needs some work. I’ll
> weigh in on the latest RC vote thread.
>
> Julian
>
>
> > On Mar 2, 2017, at 12:33 PM, Harshad Deshmukh <hars...@cs.wisc.edu>
> wrote:
> >
> > Hi Julian,
> >
> > Thanks for your inputs. I will modify the report accordingly and send it
> out for review.
> >
> > Just an update on the release progress - Marc floated a vote for RC5
> (yesterday evening CST) on the dev list, which has gotten more than 3 +1s
> so far. It appears that this release candidate (RC5) should be the final
> one, therefore I thought this week should be a realistic target for the
> release. Marc should have an idea about the next steps, so if I
> misunderstood something w.r.t the release estimates, please let me know.
> >
> >
> > On 03/02/2017 02:12 PM, Marc Spehlmann wrote:
> >> Hi Julian,
> >>
> >> I agree that 'this week' is optimistic. Though, we have been going
> through
> >> the voting process on @dev, it's likely that the current RC will not
> pass,
> >> meaning that we'll need to restart the whole process, as you said.
> >>
> >> Considering, 'this month' is probably a more realistic wording.
> >>
> >> On Thu, Mar 2, 2017 at 2:08 PM, Julian Hyde <jh...@apache.org> wrote:
> >>
> >>> In answer to the “How does the podling rate their maturity?”. The
> >>> incubator report used to have categories "Ready to Graduate", “Some
> >>> Community Growth", "No Release" and "Still Getting Started.” Out of
> these,
> >>> I think “No release” fits best. You can add that the first release is
> well
> >>> under way.
> >>>
> >>> "We will have our first release this week” is a bit optimistic.
> Remember
> >>> you need to create a release candidate, pass a vote on the Quickstep
> list
> >>> (which takes 3 working days), repeat if there is a problem with the
> RC, and
> >>> then pass a vote on the incubator list (which takes 3 working days).
> Again,
> >>> the incubator vote might fail, and in fact probably will the first or
> >>> second time.
> >>>
> >>> A true statement is “we are almost ready to start a vote on our first
> >>> release candidate”.
> >>>
> >>> Julian
> >>>
> >>>
> >>>> On Mar 2, 2017, at 11:04 AM, Harshad Deshmukh <hars...@cs.wisc.edu>
> >>> wrote:
> >>>> Hi Roman,
> >>>>
> >>>> Apologies for the delay. I am not sure how to address two of these
> >>> questions, hence left them blank. Any inputs in that regard should be
> >>> helpful. Can you please review the following report?
> >>>> *   Your project name
> >>>> Apache (incubating) Quickstep
> >>>>
> >>>> *   A brief description of your project, which assumes no knowledge of
> >>>>the project or necessarily of its field
> >>>> Apache Quickstep is high-performance database engine designed to
> exploit
> >>> the full potential
> >>>> of hardware that is packed in modern computing boxes (servers and
> >>> laptops).
> >>>> The initial version targets single-node in-memory environments.
> >>>>
> >>>> *   A list of the three most important issues to address in the move
> >>>>towards graduation.
> >>>> 1) Building a Quickstep community
> >>>> 2) More adoption of the Quickstep technology
> >>>> 3) Making the technology easier to understand and use
> >>>>
> >>>> *   Any issues that the Incubator PMC or ASF Board might wish/need to
> be
> >>>>aware of
> >>>> None
> >>>>
> >>>> *   How has the community developed since the last report
> >>>>
> >>>> *   How has the project developed since the last report.
> >>>>
> >>>> We will have our first release this week. Since the last report,
> >>>>
> >>>> 1) We have made several changes to the code base.
> >>>> Some highlights are: Cleaning up the third party library code as per
> the
> >>> Apache hygiene, improve the code performance by adding several novel
> >>> features.
> >>>> 2) Preparation for release -
> >>>> Created scripts and step by step procedural documentation for how to
> >>> make a Quickstep release. Scripts have been added to the main repo
> while
> >>> documentation is on confluence.
> >>>> We went through several release candidates. During this period, we
> >>> identified some usability issues on our supported platforms and fixed
> them.
> >>> Quickstep is now on schedule for a release later this week.
> >>>> *   How does the podling rate their own maturity.
> >>>>
> >>>>
> >>>> On 02/28/2017 10:23 PM, Roman Shaposhnik wrote:
> >>>>> Bueller? Bueller? Bueller?
> >>>> --
> >>>> Thanks,
> >>>> Harshad
> >>>>
> >>>
> >
> > --
> > Thanks,
> > Harshad
> >
>
>

Re: We really need to get going with the podling report

[Marc Spehlmann]

Hi Julian,

I agree that 'this week' is optimistic. Though, we have been going through
the voting process on @dev, it's likely that the current RC will not pass,
meaning that we'll need to restart the whole process, as you said.

Considering, 'this month' is probably a more realistic wording.

On Thu, Mar 2, 2017 at 2:08 PM, Julian Hyde  wrote:

> In answer to the “How does the podling rate their maturity?”. The
> incubator report used to have categories "Ready to Graduate", “Some
> Community Growth", "No Release" and "Still Getting Started.” Out of these,
> I think “No release” fits best. You can add that the first release is well
> under way.
>
> "We will have our first release this week” is a bit optimistic. Remember
> you need to create a release candidate, pass a vote on the Quickstep list
> (which takes 3 working days), repeat if there is a problem with the RC, and
> then pass a vote on the incubator list (which takes 3 working days). Again,
> the incubator vote might fail, and in fact probably will the first or
> second time.
>
> A true statement is “we are almost ready to start a vote on our first
> release candidate”.
>
> Julian
>
>
> > On Mar 2, 2017, at 11:04 AM, Harshad Deshmukh 
> wrote:
> >
> > Hi Roman,
> >
> > Apologies for the delay. I am not sure how to address two of these
> questions, hence left them blank. Any inputs in that regard should be
> helpful. Can you please review the following report?
> >
> > *   Your project name
> > Apache (incubating) Quickstep
> >
> > *   A brief description of your project, which assumes no knowledge of
> >the project or necessarily of its field
> > Apache Quickstep is high-performance database engine designed to exploit
> the full potential
> > of hardware that is packed in modern computing boxes (servers and
> laptops).
> > The initial version targets single-node in-memory environments.
> >
> > *   A list of the three most important issues to address in the move
> >towards graduation.
> > 1) Building a Quickstep community
> > 2) More adoption of the Quickstep technology
> > 3) Making the technology easier to understand and use
> >
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> >aware of
> > None
> >
> > *   How has the community developed since the last report
> >
> > *   How has the project developed since the last report.
> >
> > We will have our first release this week. Since the last report,
> >
> > 1) We have made several changes to the code base.
> > Some highlights are: Cleaning up the third party library code as per the
> Apache hygiene, improve the code performance by adding several novel
> features.
> >
> > 2) Preparation for release -
> > Created scripts and step by step procedural documentation for how to
> make a Quickstep release. Scripts have been added to the main repo while
> documentation is on confluence.
> > We went through several release candidates. During this period, we
> identified some usability issues on our supported platforms and fixed them.
> Quickstep is now on schedule for a release later this week.
> >
> > *   How does the podling rate their own maturity.
> >
> >
> > On 02/28/2017 10:23 PM, Roman Shaposhnik wrote:
> >> Bueller? Bueller? Bueller?
> >
> > --
> > Thanks,
> > Harshad
> >
>
>

[VOTE] incubator-quickstep-0.1.0RC5

[Marc Spehlmann]

Jianqiao and I worked today to merge some PRs that we thought should go in
the first release. Accordingly we have created another candidate.

As per Apache, *we require 3 +1 votes* from project members to make this an
official release.

Before voting, please test the release (build, run ctest). For a guide on
how to test, GOTO release/README.md

This vote will remain open for 72 hours or until we find issues that we can
quickly correct and create another candidate.

more details on

https://cwiki.apache.org/confluence/display/QUICKSTEP/How+To+Release

--Marc

[VOTE] quickstep-0.1.0rc4

[Marc Spehlmann]

Hi everyone,

This past week I took a couple steps forward in the release process and
therefore made another release candidate.

- Created a Dockerfile with an ubuntu 16 version with all* the dependencies
quickstep should need to build. This should make verification of the
release candidate easy if you're on an unsupported system (mac).

- Started work on a guide on confluence
https://cwiki.apache.org/confluence/display/QUICKSTEP/How+To+Release

- Fixed a third party bug introduced during our move to downloadable
dependencies last month

If no one has objections, I will create a folder in the root of quickstep
called release/ where we can keep the dockerfile, and release building
scripts.

Please vote +1 if you think that b24349cc3 is a good commit to base the
release on, and that you want this to be the release commit.

--Marc

* it seems that gcc 5.4 runs into an warning->error in gflags in Docker
ubuntu 16. Clang does not have this problem and it compiles fine.

[DISCUSSION] Prereqs for Quickstep

[Marc Spehlmann]

Jignesh and I spoke yesterday about how to specify what the minimum
requirements are for building quickstep. We decided to 'officially' support
ubuntu 14 LTS, 16 LTS.

Building on a mac will not be officially supported in the initial release.

Now there's the matter of what system packages will be required to build
quickstep. I'm not sure how we can specify this information. One way I
thought of was to use a Dockerfile. From that you can either
build/instantiate the image and run in a pristine environment, or you could
just look at the packages in the file and install the missing ones manually.

However, we already have the vagrant files. Does anyone use these? I'm
partial to docker because it's lightwieght.

What do people think about the issue of system prerequisites?

--Marc

0.1.0 RC3

[Marc Spehlmann]

Release Candidate 3 should now include 3rd party protobuf correctly. I've
tested it on my mac and it works correctly. Here is a handy script [1] I
have been putting together to do the release. It's still in alpha, but
the test_rc() function will do everything you need it to.

***I did not make this a vote because I'm just looking for feedback. If you
try compiling it, and it fails, please respond to dev with the error.***

For example, I tried running this on a CSL machine, and what I got was a
cmake error:

CMake Error at types/port/CMakeLists.txt:34 (message):
  Unable to find a reentrant (threadsafe) version of gmtime (tried POSIX
  gmtime_r and Windows gmtime_s).

I also tried it on my mac, and it compiles successfully.

[1] https://gist.github.com/cramja/3f0ace3b60f06dfe82b572f50faa8bf3

ps to use the handy script:

cd /tmp
mkdir qs
cd qs
curl
https://gist.githubusercontent.com/cramja/3f0ace3b60f06dfe82b572f50faa8bf3/raw/c37a1becc56cde20fddf5628ea7680b3e0137a26/make-release.sh
> make-release.sh
source make-release.sh
test_rc

[VOTE] 0.1.0 RC2

[Marc Spehlmann]

Per the instructions of impala [1], I have been stepping through the
release process. We voted once already for RC0, but it was turned down
because it did not include protobuf.

Try testing out the release candidate by pulling the SVN repo, and building
quickstep fresh, plus testing the hashes + signature. There are
instructions on how to do all of this on [1], just substitute impala with
quickstep. We're on step 15.

You can start with

  svn co https://dist.apache.org/repos/dist/dev/incubator/quickstep
quickstep-svn-dev
  cd quickstep-svn-dev/0.1.0/RC2
  tar -xzf apache-quickstep-incubating-0.1.0.tar.gz
  cd apache-quickstep-incubating-0.1.0
  # add the KEYS file from quickstep to your gpg key ring
  gpg --verify apache-quickstep-incubating-0.1.0.tar.gz.asc
  d5sum --check apache-quickstep-incubating-0.1.0.tar.gz.md5
  sha1sum --check apache-quickstep-incubating-0.1.0.tar.gz.sha
  # then try building quickstep, first pull third_party, then do your
normal build

Once you try this, please +1/-1 this vote based on your result.

--Marc

[1]  https://cwiki.apache.org/confluence/display/IMPALA/
DRAFT%3A+How+to+Release

[VOTE] Quickstep-0.1.0 RC0

[Marc Spehlmann]

Per the instructions of impala [1], I have been stepping through the
release process. Every time there is a release there can be multiple
release candidates. We vote on /dev as to whether we agree that we agree
this is worthy of a release.

So, if you please, try testing out the release candidate by pulling the SVN
repo, and building quickstep fresh, plus testing the hashes + signature.
There are instructions on how to do all of this on [1], just substitute
impala with quickstep. We're on step 15.

You can start with

  svn co https://dist.apache.org/repos/dist/dev/incubator/quickstep

(Note, the sig should not work, I haven't added myself to KEYS, but I will
in a minute here, please then don't vote +1 on this release candidate,
we'll try another tomorrow.)


[1]
https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release

Re: release candidate svn repo

[Marc Spehlmann]

Awesome, thank you so much! Is creating a directory in svn on
dist.apache.org something which is in our power as apache members?

On Tue, Feb 14, 2017 at 4:24 PM, Julian Hyde <jh...@apache.org> wrote:

> I've just created the quickstep directory in svn. Now if you're a
> committer, you should be able to do this:
>
>   svn co https://dist.apache.org/repos/dist/dev/incubator/quickstep
>
>
> Julian
>
>
> On Tue, Feb 14, 2017 at 2:12 PM, Marc Spehlmann <sp...@apache.org> wrote:
> > I'm looking through the impala guide to releases and they have an svm
> repo
> > where they store their release candidate tarballs, hashes plus an svn for
> > the final releases.
> >
> > https://cwiki.apache.org/confluence/display/IMPALA/
> DRAFT%3A+How+to+Release
> >
> > repo is
> >
> > svn checkout https://dist.apache.org/repos/dist/dev/incubator/impala/
> >
> > Anyone know how to make an svm repo on apache.org?
>

release candidate svn repo

[Marc Spehlmann]

I'm looking through the impala guide to releases and they have an svm repo
where they store their release candidate tarballs, hashes plus an svn for
the final releases.

https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release

repo is

svn checkout https://dist.apache.org/repos/dist/dev/incubator/impala/

Anyone know how to make an svm repo on apache.org?

Re: Editing documentation for release: BUILDING.md

[Marc Spehlmann]

curl is needed for the download_script. There's probably many other prereqs
(git/bash) that are required, but they're sort of obvious to system
builders. So I could see removing it for that reason.

On Tue, Feb 7, 2017 at 6:51 PM, Jignesh Patel <jmp.quicks...@gmail.com>
wrote:

> Marc,
>
> Should we get rid of curl from the build instructions? We don’t need it
> with the current instruction flow.
>
> Cheers,
> Jignesh
>
> On 2/7/17, 6:43 PM, "Marc Spehlmann" <spehl.apa...@gmail.com> wrote:
>
> For example, here is the edited doc:
> https://github.com/cramja/incubator-quickstep/blob/
> refactor-building/BUILDING.md#prerequisites
>
> On Tue, Feb 7, 2017 at 6:28 PM, Marc Spehlmann <spehl.apa...@gmail.com
> >
> wrote:
>
> > Any objections if I remove the 'Getting cmake' section? It seems
> long, and
> > like information that a database programmer would probably already
> > know/google and quickly find out.
> >
>     > Instead I can link to cmake's webpage in the prereq section
> >
> > On Tue, Feb 7, 2017 at 6:25 PM, Marc Spehlmann <
> spehl.apa...@gmail.com>
> > wrote:
> >
> >> Re-reading BUILDING.md, it seems like this doc might be a victim of
> being
> >> added to iteratively over time without consideration to the overall
> >> structure. I am purposing to par down some of the language and
> shift the
> >> sections around in a flow that a novice quickstep user would
> probably want:
> >>
> >> 0 a header with TOC
> >> 1 basic building instructions
> >> 2 common flags we use for quick troubleshooting
> >> 3 List of advanced cmake flags
> >> 4 appendix of other, uncommon features
> >>- building on windows
> >>- vagrant
> >>
> >> I'll make a PR with these changes, but does anyone have an initial
> >> objection to the change in structure?
> >>
> >
> >
>
>
>
>

Re: Editing documentation for release: BUILDING.md

[Marc Spehlmann]

For example, here is the edited doc:
https://github.com/cramja/incubator-quickstep/blob/refactor-building/BUILDING.md#prerequisites

On Tue, Feb 7, 2017 at 6:28 PM, Marc Spehlmann <spehl.apa...@gmail.com>
wrote:

> Any objections if I remove the 'Getting cmake' section? It seems long, and
> like information that a database programmer would probably already
> know/google and quickly find out.
>
> Instead I can link to cmake's webpage in the prereq section
>
> On Tue, Feb 7, 2017 at 6:25 PM, Marc Spehlmann <spehl.apa...@gmail.com>
> wrote:
>
>> Re-reading BUILDING.md, it seems like this doc might be a victim of being
>> added to iteratively over time without consideration to the overall
>> structure. I am purposing to par down some of the language and shift the
>> sections around in a flow that a novice quickstep user would probably want:
>>
>> 0 a header with TOC
>> 1 basic building instructions
>> 2 common flags we use for quick troubleshooting
>> 3 List of advanced cmake flags
>> 4 appendix of other, uncommon features
>>- building on windows
>>- vagrant
>>
>> I'll make a PR with these changes, but does anyone have an initial
>> objection to the change in structure?
>>
>
>

Re: Editing documentation for release: BUILDING.md

[Marc Spehlmann]

Any objections if I remove the 'Getting cmake' section? It seems long, and
like information that a database programmer would probably already
know/google and quickly find out.

Instead I can link to cmake's webpage in the prereq section

On Tue, Feb 7, 2017 at 6:25 PM, Marc Spehlmann <spehl.apa...@gmail.com>
wrote:

> Re-reading BUILDING.md, it seems like this doc might be a victim of being
> added to iteratively over time without consideration to the overall
> structure. I am purposing to par down some of the language and shift the
> sections around in a flow that a novice quickstep user would probably want:
>
> 0 a header with TOC
> 1 basic building instructions
> 2 common flags we use for quick troubleshooting
> 3 List of advanced cmake flags
> 4 appendix of other, uncommon features
>- building on windows
>- vagrant
>
> I'll make a PR with these changes, but does anyone have an initial
> objection to the change in structure?
>

Re: Release Signing

[Marc Spehlmann]

Thank you for clearing that up.

We need to select a single release manager.

On Wed, Feb 1, 2017 at 9:26 PM, Julian Hyde <jh...@apache.org> wrote:

> Yes, only the release manager needs to sign.
>
> The KEYS file contains all the people who have EVER signed a release.
>
> > On Feb 1, 2017, at 6:20 PM, Jignesh Patel <jmp.quicks...@gmail.com>
> wrote:
> >
> > That would be nice if that were the case. We are nearly there then!
> >
> > Cheers,
> > Jignesh
> >
> > On 2/1/17, 7:47 PM, "Marc Spehlmann" <spehl.apa...@gmail.com> wrote:
> >
> >It sounds like only one release manager needs to sign it then?
> >
> >
> >
> >
>
>

Re: Release Signing

[Marc Spehlmann]

Oh that was one thing I was confused about was the number of signers
needed. I took it that all the release managers need to sign, and that
there are several release managers.

Each manager would need to their private key to the signing process,
something which could only be done by passing the tarball around to
people's private laptop. Obviously, that's not efficient.

It sounds like only one release manager needs to sign it then?

On Tue, Jan 31, 2017 at 6:14 PM, Julian Hyde <jh...@apache.org> wrote:

> It does say so in the instructions, but I’ll reiterate: be sure to use
> your apache.org <http://apache.org/> email address for your key. People
> get spooked if they get a release that is not signed by someone who is not
> obviously an Apache committer.
>
> Generally the release manager will either build the release on their own
> machine or download a build to their machine. Then they will sign it on
> their machine (where their private key is present). Lastly they will upload
> it (which happens by means of a “svn commit”).
>
> At the same time they will make sure that their key is in KEYS, and if not
> they will edit KEYS and do another “svn commit”.
>
> Julian
>
>
>
>
> > On Jan 31, 2017, at 3:35 PM, Marc Spehlmann <spehl.apa...@gmail.com>
> wrote:
> >
> > One of the steps that must take place before releasing a release tarball
> is
> > to have the release managers digitally sign the tarball.
> >
> > Hakan, Jignesh, Harshad I think you all are the release managers. Please
> > follow this guide
> >
> > http://quickstep.apache.org/release-signing/
> >
> > to
> > 1) create a key pair
> > 2) upload the public key to a public keyserver
> > 3) (bonus for now) add the public key to a KEYS file in the root of
> > quickstep.
> >
> > When the release tarball is ready, we can sign it.
> >
> > To be fair, I'm not totally sure how this works because it seems to me
> that
> > everyone has to sign the release with their private key, meaning that it
> > must be uploaded to each PC where the private key is held, then signed?
> > That seems cumbersome.
> >
> > Anyways, steps 1,2 are straightforward and need to be done before we
> > resolve that last problem.
> >
> > Cheers,
> > Marc
>
>

Release Signing

[Marc Spehlmann]

One of the steps that must take place before releasing a release tarball is
to have the release managers digitally sign the tarball.

Hakan, Jignesh, Harshad I think you all are the release managers. Please
follow this guide

http://quickstep.apache.org/release-signing/

to
1) create a key pair
2) upload the public key to a public keyserver
3) (bonus for now) add the public key to a KEYS file in the root of
quickstep.

When the release tarball is ready, we can sign it.

To be fair, I'm not totally sure how this works because it seems to me that
everyone has to sign the release with their private key, meaning that it
must be uploaded to each PC where the private key is held, then signed?
That seems cumbersome.

Anyways, steps 1,2 are straightforward and need to be done before we
resolve that last problem.

Cheers,
Marc

Re: release: third_party/

[Marc Spehlmann]

Also, IWYU script was written by Shoban

On Mon, Jan 9, 2017 at 3:39 PM, Marc Spehlmann <spehl.apa...@gmail.com>
wrote:

> Thanks for finding these! Impala is more or less doing what we're doing,
> except it doesn't use submodules. One thing is that if you download their
> release (http://impala.apache.org/downloads.html) it's not clear where
> all the thirdparty dependencies went.
>
> On Mon, Jan 9, 2017 at 12:00 PM, Hakan Memisoglu <
> hakanmemiso...@apache.org> wrote:
>
>> Hi Marc,
>>
>> Here are two Apache projects using C++:
>>
>> Impala: https://github.com/cloudera/Impala <https://github.com/cloudera/I
>> mpala>
>>
>> Kudu: https://github.com/cloudera/kudu <https://github.com/cloudera/kudu>
>>
>> Kudu has a download script for dependencies in third_party folder.
>> It seems to be a good example for our case.
>>
>> > On Jan 8, 2017, at 2:17 PM, Marc Spehlmann <spehl.apa...@gmail.com>
>> wrote:
>> >
>> > Regarding the IWYU script. It sounds safe then. If anything, maybe it
>> > belongs in another folder. ("tools/" ?)
>> >
>> > On Thu, Jan 5, 2017 at 3:07 PM, Julian Hyde <jh...@apache.org> wrote:
>> >
>> >> Better still, make a release depend only releases. Snapshots and even
>> >> commit IDs can disappear.
>> >>
>> >> And consider the state of the IP. In Apache and many other open source
>> >> projects, the IP is only guaranteed "clean" when it has been formally
>> >> released.
>> >>
>> >> Sorry this is all painful. But conversely, this is the value-add of
>> >> making a release.
>> >>
>> >> On Thu, Jan 5, 2017 at 11:57 AM, Harshad Deshmukh <hars...@cs.wisc.edu
>> >
>> >> wrote:
>> >>>> Also, since you’re relying on the latest googletest, protobuf and re2
>> >> your
>> >>>> release will work today and will break at some point in the future.
>> >> That’s
>> >>>> not OK. A release can depend only on releases, not snapshots or live
>> >>>> repositories.
>> >>>
>> >>> Thanks for pointing that. It is possible to fetch a particular release
>> >>> (based on its commit ID) of the submodule. We will add the release
>> >> commit ID
>> >>> to the git submodule file.
>> >>>
>> >>>
>> >>>
>> >>> On 01/05/2017 01:44 PM, Julian Hyde wrote:
>> >>>>
>> >>>> Git sub-modules are useful, I agree. But for a source release, the
>> goal
>> >> is
>> >>>> that someone should be able to download the source tar-ball and go:
>> >>>>
>> >>>> $ curl -O …/apache-quickstep-x.x-incubating.tar.gz
>> >>>> $ tar xvfz apache-quickstep-x.x-incubating.tar.gz
>> >>>> $ cd apache-quickstep-x.x-incubating/build
>> >>>> $ ./download-thirdparty.sh
>> >>>> $ cmake etc.
>> >>>>
>> >>>> I tried to find other Apache projects that use git submodules and see
>> >> what
>> >>>> they do for source releases. Not much luck. Maybe someone else can
>> find
>> >>>> something.
>> >>>>
>> >>>> Also, since you’re relying on the latest googletest, protobuf and re2
>> >> your
>> >>>> release will work today and will break at some point in the future.
>> >> That’s
>> >>>> not OK. A release can depend only on releases, not snapshots or live
>> >>>> repositories.
>> >>>>
>> >>>> Julian
>> >>>>
>> >>>>
>> >>>>
>> >>>>> On Jan 5, 2017, at 11:03 AM, Harshad Deshmukh <hars...@cs.wisc.edu>
>> >>>>> wrote:
>> >>>>>
>> >>>>> Thanks for the review Julian.
>> >>>>>
>> >>>>> For some of the third party libraries used in Quickstep (e.g.
>> >> googletest,
>> >>>>> protobuf and re2) we use the submodules feature of git. For such
>> >> libraries,
>> >>>>> the developer has to initialize the submodule only once, which pulls
>> >> code
>> >>>>> from the third party repo to the Quickstep third party directory.
>> >>>>>
>> >>>>> I don't know of a centra

Re: release: third_party/

[Marc Spehlmann]

Thanks for finding these! Impala is more or less doing what we're doing,
except it doesn't use submodules. One thing is that if you download their
release (http://impala.apache.org/downloads.html) it's not clear where all
the thirdparty dependencies went.

On Mon, Jan 9, 2017 at 12:00 PM, Hakan Memisoglu <hakanmemiso...@apache.org>
wrote:

> Hi Marc,
>
> Here are two Apache projects using C++:
>
> Impala: https://github.com/cloudera/Impala <https://github.com/cloudera/
> Impala>
>
> Kudu: https://github.com/cloudera/kudu <https://github.com/cloudera/kudu>
>
> Kudu has a download script for dependencies in third_party folder.
> It seems to be a good example for our case.
>
> > On Jan 8, 2017, at 2:17 PM, Marc Spehlmann <spehl.apa...@gmail.com>
> wrote:
> >
> > Regarding the IWYU script. It sounds safe then. If anything, maybe it
> > belongs in another folder. ("tools/" ?)
> >
> > On Thu, Jan 5, 2017 at 3:07 PM, Julian Hyde <jh...@apache.org> wrote:
> >
> >> Better still, make a release depend only releases. Snapshots and even
> >> commit IDs can disappear.
> >>
> >> And consider the state of the IP. In Apache and many other open source
> >> projects, the IP is only guaranteed "clean" when it has been formally
> >> released.
> >>
> >> Sorry this is all painful. But conversely, this is the value-add of
> >> making a release.
> >>
> >> On Thu, Jan 5, 2017 at 11:57 AM, Harshad Deshmukh <hars...@cs.wisc.edu>
> >> wrote:
> >>>> Also, since you’re relying on the latest googletest, protobuf and re2
> >> your
> >>>> release will work today and will break at some point in the future.
> >> That’s
> >>>> not OK. A release can depend only on releases, not snapshots or live
> >>>> repositories.
> >>>
> >>> Thanks for pointing that. It is possible to fetch a particular release
> >>> (based on its commit ID) of the submodule. We will add the release
> >> commit ID
> >>> to the git submodule file.
> >>>
> >>>
> >>>
> >>> On 01/05/2017 01:44 PM, Julian Hyde wrote:
> >>>>
> >>>> Git sub-modules are useful, I agree. But for a source release, the
> goal
> >> is
> >>>> that someone should be able to download the source tar-ball and go:
> >>>>
> >>>> $ curl -O …/apache-quickstep-x.x-incubating.tar.gz
> >>>> $ tar xvfz apache-quickstep-x.x-incubating.tar.gz
> >>>> $ cd apache-quickstep-x.x-incubating/build
> >>>> $ ./download-thirdparty.sh
> >>>> $ cmake etc.
> >>>>
> >>>> I tried to find other Apache projects that use git submodules and see
> >> what
> >>>> they do for source releases. Not much luck. Maybe someone else can
> find
> >>>> something.
> >>>>
> >>>> Also, since you’re relying on the latest googletest, protobuf and re2
> >> your
> >>>> release will work today and will break at some point in the future.
> >> That’s
> >>>> not OK. A release can depend only on releases, not snapshots or live
> >>>> repositories.
> >>>>
> >>>> Julian
> >>>>
> >>>>
> >>>>
> >>>>> On Jan 5, 2017, at 11:03 AM, Harshad Deshmukh <hars...@cs.wisc.edu>
> >>>>> wrote:
> >>>>>
> >>>>> Thanks for the review Julian.
> >>>>>
> >>>>> For some of the third party libraries used in Quickstep (e.g.
> >> googletest,
> >>>>> protobuf and re2) we use the submodules feature of git. For such
> >> libraries,
> >>>>> the developer has to initialize the submodule only once, which pulls
> >> code
> >>>>> from the third party repo to the Quickstep third party directory.
> >>>>>
> >>>>> I don't know of a centralized repo for C++ projects. Does the git
> >>>>> submodule method sound similar to the maven central approach you
> >> mentioned?
> >>>>>
> >>>>> On 01/05/2017 12:43 PM, Julian Hyde wrote:
> >>>>>>
> >>>>>> I took a quick look at third_party and there don’t seem to be any
> >>>>>> binaries in there. That’s good. You definitely cannot include
> >> binaries in a
> >>>>>> sour

Re: release: third_party/

[Marc Spehlmann]

That seems to be the repo with the LLVM code for implementing IWYU. I think
what we have in our repo is scripts ontop of that library. I'm wondering
where the scripts came from.

Thanks,
Marc

On Thu, Jan 5, 2017 at 10:50 AM, Harshad Deshmukh <hars...@cs.wisc.edu>
wrote:

> Hi Marc,
>
> How about this one for IWYU?
>
> https://github.com/include-what-you-use/include-what-you-use
> /blob/master/LICENSE.TXT
>
>
> On 01/05/2017 10:43 AM, Marc Spehlmann wrote:
>
>> I double checked the libraries in thirdy_party. They adhere to Apache's
>> 3rd
>> party requirement as they are all apache 2 or opensourced by Google. The
>> only issue I saw was that IWYU has no documentation. Anyone know of its
>> source?
>>
>> Library
>>
>> Ver
>>
>> License
>>
>> Notes
>>
>> benchmark
>>
>> Apache 2.0
>>
>> cpplint
>>
>> Google
>>
>> Header states that reuse is unconditional so long as the copyright header
>> stays intact.
>>
>> Farmhash
>>
>> Google
>>
>> No restrictions so long as COPYING file is preserved. See COPYING
>>
>> gflags
>>
>> Google
>>
>> No restrictions so long as COPYING file is preserved. See COPYING
>>
>> glog
>>
>> Google
>>
>> No restrictions so long as COPYING file is preserved. See COPYING
>>
>> gtest
>>
>> Google
>>
>> No restrictions so long as COPYING file is preserved. See COPYING in
>> subprojects.
>>
>> gperftools
>>
>> Google
>>
>> No restrictions so long as COPYING file is preserved.
>>
>> iwyu
>>
>>
>> No license present
>>
>> linenoise
>>
>> Google
>>
>> No restrictions so long as LICENSE file is preserved.
>>
>> protobuf
>>
>> Google
>>
>> No restrictions so long as LICENSE file is preserved.
>>
>> RE2
>>
>> Google
>>
>> No restrictions so long as LICENSE file is preserved.
>>
>> tmb
>>
>> Apache 2.0
>>
>> README: TMB is part of the Quickstep project (copyright Pivotal Software,
>> Inc.) and is distributed under the same license terms.
>>
>>
> --
> Thanks,
> Harshad
>
>

Re: blog

[Marc Spehlmann]

Fixed - I followed the asterixdb guys are doing and committed the compiled
site, building it in the content directory.

On Tue, Dec 13, 2016 at 11:27 AM, Julian Hyde <jh...@apache.org> wrote:

> Correct. The site (and the publishing process) don't do anything clever.
>
> On Tue, Dec 13, 2016 at 8:39 AM, Marc Spehlmann <spehl.apa...@gmail.com>
> wrote:
> > Whoops, looks like the content was pushed OK, but the mechanism which is
> > supposed to trigger Jekyll compilation was not tripped- we just see a
> > directory listing when going to http://quickstep.apache.org/.
> >
> > Possibly the automatic jekyll publishing isn't built into this framework.
> > The instructions for https://github.com/apache/calcite/tree/master/site
> > suggest that the site must be compiled locally and then pushed into the
> > pubsub svn repo.
> >
> > If that's the case, then maybe we will have to compile the jekyll files
> > locally and push them along with the markdown/config/scss/etc
> >
> > On Tue, Dec 13, 2016 at 10:10 AM, Marc Spehlmann <spehl.apa...@gmail.com
> >
> > wrote:
> >
> >> Harshad pointed out the directory is
> >> https://git-wip-us.apache.org/repos/asf?p=incubator-quickstep-site.git
> >>
> >> He found it by modifying the address found in
> >> https://cwiki.apache.org/confluence/display/QUICKSTEP/
> >> Workflow+For+Committers
> >>
> >> On Tue, Dec 13, 2016 at 10:02 AM, Marc Spehlmann <
> spehl.apa...@gmail.com>
> >> wrote:
> >>
> >>> Thanks for looking into that! I actually took the majority of the
> written
> >>> content from the old website and converted it into markdown to create
> the
> >>> content for the new site, so I think we can keep the structure as
> Jekyll
> >>> has automatically made it. Once we get some more posts, we might want
> to
> >>> move to a separate /blog directory for the posts and have it paginate
> the
> >>> blog index- that's not too hard to do, but unneeded right now.
> >>>
> >>> One question about the quickstep-site repo...
> >>>
> >>> Any idea how to get access? Or will I need to submit a pull request?
> >>>
> >>> cramja@wifi-780 ~/workspace/incubator-quickstep-site $ git remote -v
> >>> origin https://github.com/cramja/incubator-quickstep-site.git (fetch)
> >>> origin https://github.com/cramja/incubator-quickstep-site.git (push)
> >>> source https://github.com/apache/incubator-quickstep-site (fetch)
> >>> source https://github.com/apache/incubator-quickstep-site (push)
> >>> cramja@wifi-780 ~/workspace/incubator-quickstep-site $ git push source
> >>> remote: Permission to apache/incubator-quickstep-site.git denied to
> >>> cramja.
> >>> fatal: unable to access 'https://github.com/apache/inc
> >>> ubator-quickstep-site/': The requested URL returned error: 403
> >>>
> >>> On Mon, Dec 12, 2016 at 5:24 PM, Roman Shaposhnik <
> ro...@shaposhnik.org>
> >>> wrote:
> >>>
> >>>> Yes. But note you have to push to a branch called asf-site.
> >>>>
> >>>> Thanks,
> >>>> Roman.
> >>>>
> >>>> On Mon, Dec 12, 2016 at 1:14 PM, Julian Hyde <jh...@apache.org>
> wrote:
> >>>> > INFRA tell me that Quickstep is already using gitpubsub. So, if you
> >>>> push to https://github.com/apache/incubator-quickstep-site <
> >>>> https://github.com/apache/incubator-quickstep-site> it will
> >>>> automatically appear at http://quickstep.apache.org/ <
> >>>> http://quickstep.apache.org/>.
> >>>> >
> >>>> > I think you should consider merging the content you just created
> into
> >>>> that site (maybe under http://quickstep.apache.org/blog <
> >>>> http://quickstep.apache.org/blog> ?), and them turn off
> apache.github.io.
> >>>> I can help with the Jekyll if you can’t figure it out.
> >>>> >
> >>>> > Julian
> >>>> >
> >>>> >
> >>>> >
> >>>> >> On Dec 12, 2016, at 10:58 AM, Julian Hyde <jh...@apache.org>
> wrote:
> >>>> >>
> >>>> >> I got part way through the process and logged
> >>>> https://issues.apache.org/jira/browse/INFRA-13079 <
> >>>> https://issues.apache.org/jira/browse/INFRA-13079> for the rest.
> Ple

Re: blog

[Marc Spehlmann]

Whoops, looks like the content was pushed OK, but the mechanism which is
supposed to trigger Jekyll compilation was not tripped- we just see a
directory listing when going to http://quickstep.apache.org/.

Possibly the automatic jekyll publishing isn't built into this framework.
The instructions for https://github.com/apache/calcite/tree/master/site
suggest that the site must be compiled locally and then pushed into the
pubsub svn repo.

If that's the case, then maybe we will have to compile the jekyll files
locally and push them along with the markdown/config/scss/etc

On Tue, Dec 13, 2016 at 10:10 AM, Marc Spehlmann <spehl.apa...@gmail.com>
wrote:

> Harshad pointed out the directory is
> https://git-wip-us.apache.org/repos/asf?p=incubator-quickstep-site.git
>
> He found it by modifying the address found in
> https://cwiki.apache.org/confluence/display/QUICKSTEP/
> Workflow+For+Committers
>
> On Tue, Dec 13, 2016 at 10:02 AM, Marc Spehlmann <spehl.apa...@gmail.com>
> wrote:
>
>> Thanks for looking into that! I actually took the majority of the written
>> content from the old website and converted it into markdown to create the
>> content for the new site, so I think we can keep the structure as Jekyll
>> has automatically made it. Once we get some more posts, we might want to
>> move to a separate /blog directory for the posts and have it paginate the
>> blog index- that's not too hard to do, but unneeded right now.
>>
>> One question about the quickstep-site repo...
>>
>> Any idea how to get access? Or will I need to submit a pull request?
>>
>> cramja@wifi-780 ~/workspace/incubator-quickstep-site $ git remote -v
>> origin https://github.com/cramja/incubator-quickstep-site.git (fetch)
>> origin https://github.com/cramja/incubator-quickstep-site.git (push)
>> source https://github.com/apache/incubator-quickstep-site (fetch)
>> source https://github.com/apache/incubator-quickstep-site (push)
>> cramja@wifi-780 ~/workspace/incubator-quickstep-site $ git push source
>> remote: Permission to apache/incubator-quickstep-site.git denied to
>> cramja.
>> fatal: unable to access 'https://github.com/apache/inc
>> ubator-quickstep-site/': The requested URL returned error: 403
>>
>> On Mon, Dec 12, 2016 at 5:24 PM, Roman Shaposhnik <ro...@shaposhnik.org>
>> wrote:
>>
>>> Yes. But note you have to push to a branch called asf-site.
>>>
>>> Thanks,
>>> Roman.
>>>
>>> On Mon, Dec 12, 2016 at 1:14 PM, Julian Hyde <jh...@apache.org> wrote:
>>> > INFRA tell me that Quickstep is already using gitpubsub. So, if you
>>> push to https://github.com/apache/incubator-quickstep-site <
>>> https://github.com/apache/incubator-quickstep-site> it will
>>> automatically appear at http://quickstep.apache.org/ <
>>> http://quickstep.apache.org/>.
>>> >
>>> > I think you should consider merging the content you just created into
>>> that site (maybe under http://quickstep.apache.org/blog <
>>> http://quickstep.apache.org/blog> ?), and them turn off apache.github.io.
>>> I can help with the Jekyll if you can’t figure it out.
>>> >
>>> > Julian
>>> >
>>> >
>>> >
>>> >> On Dec 12, 2016, at 10:58 AM, Julian Hyde <jh...@apache.org> wrote:
>>> >>
>>> >> I got part way through the process and logged
>>> https://issues.apache.org/jira/browse/INFRA-13079 <
>>> https://issues.apache.org/jira/browse/INFRA-13079> for the rest. Please
>>> watch that case if you want to be updated.
>>> >>
>>> >>> On Dec 12, 2016, at 10:38 AM, Julian Hyde <jh...@apache.org >> jh...@apache.org>> wrote:
>>> >>>
>>> >>> The SVN URL should be https://svn.apache.org/repos/a
>>> sf/incubator/quickstep/site/ <https://svn.apache.org/repos/
>>> asf/incubator/quickstep/site/> (see for instance
>>> https://svn.apache.org/repos/asf/incubator/tephra/site/ <
>>> https://svn.apache.org/repos/asf/incubator/tephra/site/>) but doesn’t
>>> exist currently.
>>> >>>
>>> >>> Per http://incubator.apache.org/guides/mentor.html#Set+Up+SVN+Re
>>> pository <http://incubator.apache.org/guides/mentor.html#Set+Up+SVN+R
>>> epository> I think this means I need to create the repository and
>>> authorize committers. (Don’t worry, Quickstep will continue to use git for
>>> its source code, and for the source of the web site; SVN is just for the
>>> generated web site

Re: blog

[Marc Spehlmann]

Hi Julian, I'm sure that we will be!

 And in an effort to consolidate the number of quickstep pages, I'm
wondering if you'd know where the switch is that tells the DNS servers
where to point traffic to quickstep.apache.org? Right now I think it's
pointing to this (https://github.com/apache/incubator-quickstep-site)
github repo, which is obsolete now that all the info has been copied over
to quickstep's gh-pages branch.

Best,
Marc

On Sun, Dec 11, 2016 at 4:54 PM, Julian Hyde <jh...@apache.org> wrote:

> By the way, I think for now publishing to GitHub.io is just fine. The blog
> post is great — content like this is great outreach.
>
> Speaking of outreach, the ApacheQuickstep twitter account has been very
> quiet. How about tweeting a link to the blog post? I think you could find a
> couple of newsworthy events every week to tweet about.
>
> Julian
>
>
> > On Dec 11, 2016, at 2:40 PM, Julian Hyde <jh...@apache.org> wrote:
> >
> > I’m a fan of Jekyll too. Note that you can use the same markup to write
> pages and then deploy them to Apache. In fact Calcite’s web site does
> exactly that[1]. You need to install Jekyll but that isn’t too difficult.
> >
> > Julian
> >
> > [1] https://github.com/apache/calcite/blob/master/site/README.md <
> https://github.com/apache/calcite/blob/master/site/README.md>
> >
> >> On Dec 11, 2016, at 12:36 PM, Hakan Memisoglu <
> hakanmemiso...@apache.org <mailto:hakanmemiso...@apache.org>> wrote:
> >>
> >> Thanks Marc for the effort!
> >>
> >>
> >> On 12/11/2016 11:54 AM, Marc Spehlmann wrote:
> >>> Hey, we have a jekyll blog and a first blog article now! It can be
> viewed
> >>>
> >>> here http://apache.github.io/incubator-quickstep/ <
> http://apache.github.io/incubator-quickstep/>
> >>>
> >>> It's a page automatically generated by github. We can probably change
> the
> >>> URL somehow, not sure off hand though.
> >>>
> >>> Editting instructions are included in the readme here
> >>>
> >>> https://github.com/apache/incubator-quickstep/tree/gh-pages <
> https://github.com/apache/incubator-quickstep/tree/gh-pages>
> >>>
> >>> If anyone wants to touch up the css or layout, have at it. I'm no web
> >>> design expert, I rely on the pre-built themes.
> >>>
> >>> Anyone can write a post, all you need to know is markdown.
> >>>
> >>> Happy blogging!
> >>>
> >>
> >
>
>

blog

[Marc Spehlmann]

Hey, we have a jekyll blog and a first blog article now! It can be viewed

here http://apache.github.io/incubator-quickstep/

It's a page automatically generated by github. We can probably change the
URL somehow, not sure off hand though.

Editting instructions are included in the readme here

https://github.com/apache/incubator-quickstep/tree/gh-pages

If anyone wants to touch up the css or layout, have at it. I'm no web
design expert, I rely on the pre-built themes.

Anyone can write a post, all you need to know is markdown.

Happy blogging!

creating a branch on the apache repo

[Marc Spehlmann]

Hi dev, I'm trying to create a gh-pages branch on our apache repo. I added
2 upstream branches:

https://github.com/apache/incubator-quickstep.git
git://git.apache.org/incubator-quickstep.git

However, I cannot push my new branch to either of them. On the github repo,
it's clear that people are able to create branches on the apache repo, so
my question is, how do I create and push to a branch on the apache repo?

Thanks!
Marc

site

[Marc Spehlmann]

I noticed that we have a site @
https://github.com/apache/incubator-quickstep-site

and that it is hosted at

http://quickstep.incubator.apache.org/

Does anyone know/care if I modify this website?

In particular, I was thinking to make a blog using jekyll - a markdown
based blogging framework which github builds natively into each repo that
gets created. I've used jekyll before- it's great.

The site would have an index page like the current one, then a link the the
blog page which is a paginated index of blog entries. Entries would be
written by us in markdown and then the jekyll framework does the conversion
to HTML.

So if anyone has info related to this, please reply.

--Marc

Re: Working towards a release

[Marc Spehlmann]

I know that Julian mentioned writing some blog posts. I'd like to write
one. Maybe we could distill the recent paper, or talk about a specific
component? Or just talk about all the recent improvements which got our
performance way up before the paper.

On Sat, Dec 3, 2016 at 5:04 PM, Jignesh Patel 
wrote:

> Thanks Harshad for volunteering!
>
> Re. 3.5, I think the generated files are ok.
>
> We can sync up after you come back. Happy Travels.
>
> Cheers,
> Jignesh
>
> On 12/3/16, 7:25 AM, "Harshad Deshmukh"  wrote:
>
> A question on 3.5 - We have preprocessed parser and lexer files in the
> parser module. Should we remove them?
>
>
>
>
>

Re: building on OSX 10.12.1

[Marc Spehlmann]

I tried running docker with the latest ubuntu. That works, though sadly the
mounted volume feature of docker doesn't seem to work correctly, in that i
get build errors when I try to build in a shared volume.

There's also virtual box. Or just plain old ubuntu. Clion works well in
ubuntu anyways.

As for updating the module, yes, Harshad and I could chat in person next
monday.

On Wed, Nov 23, 2016 at 8:27 PM, J Patel <jmp.quicks...@gmail.com> wrote:

> I think the right way to do this is to move to the latest version of their
> libraries. Things like syscall are now deprecated in OSX.
>
> I think Zuyu and Harshad may have some insights on how to do this right.
>
> On Tue, Nov 22, 2016 at 8:49 PM Marc Spehlmann <spehl.apa...@gmail.com>
> wrote:
>
> > That link appears to be broken. Do you mind copying the text?
> >
> > On Tue, Nov 22, 2016 at 8:24 PM, Navneet Sankara <nav...@gmail.com>
> wrote:
> >
> > > Probably this macOS Sierra issue:
> > > https://lists.apache.org/thread.html/81e200d3599f519009ebe54674f4aa
> > > 96c71f3e22ccbacaf9e5198332@
> > > 
> > >
> > > On Tue, Nov 22, 2016, 19:50 Marc Spehlmann <spehl.apa...@gmail.com>
> > wrote:
> > >
> > > > Hello quickstep,
> > > > I'm getting compile errors with the third party libraries. I have
> some
> > of
> > > > these installed on my system (protobuf). Is there a flag to tell
> CMake
> > to
> > > > use the system library instead of building the bundled one?
> > > >
> > >
> >
>

Re: building on OSX 10.12.1

[Marc Spehlmann]

That link appears to be broken. Do you mind copying the text?

On Tue, Nov 22, 2016 at 8:24 PM, Navneet Sankara <nav...@gmail.com> wrote:

> Probably this macOS Sierra issue:
> https://lists.apache.org/thread.html/81e200d3599f519009ebe54674f4aa
> 96c71f3e22ccbacaf9e5198332@
> 
>
> On Tue, Nov 22, 2016, 19:50 Marc Spehlmann <spehl.apa...@gmail.com> wrote:
>
> > Hello quickstep,
> > I'm getting compile errors with the third party libraries. I have some of
> > these installed on my system (protobuf). Is there a flag to tell CMake to
> > use the system library instead of building the bundled one?
> >
>

SplitRow Now fixed

[Marc Spehlmann]

Hello, we were experiencing issues with the splitrow store which amounted
to an overestimate of the amount of tuples we could insert into the block.
This has been fixed in #125 and I tried loading SSB with splitrow, it works
fine.

The issue was appearing in SSB and not TPCH because TPCH only uses fixed
length (CHAR) attributes while SSB uses variable (VARCHAR). This brought up
another discussion of which Char type to use in the TPCH benchmark- we
might need to modify the scripts.

Best,
Marc

TPCH 100

[Marc Spehlmann]

Hey 2 questions about TPCH sf100.

(1)
On a cloud lab box, I am having trouble getting query 17 to finish. It
seems to be stuck in Hash table operations. Any ideas/experience?

(2)
Anyone got the link to the spreadsheet with benchmark results?

-Marc

compiler warnings- anyone seen this?

[Marc Spehlmann]

I get annoying compiler warnings upgraded to errors when building with
clang 3.8, bison 3.0.4, flex 2.6.0:

[3/170] Building CXX object
parser/CMakeFiles/quickstep_parser_SqlLexer.dir/SqlLexer_gen.cpp.o
FAILED: parser/CMakeFiles/quickstep_parser_SqlLexer.dir/SqlLexer_gen.cpp.o
ccache /usr/bin/clang++-3.8
-DQUICKSTEP_ENABLE_VECTOR_COPY_ELISION_SELECTION
-DQUICKSTEP_ENABLE_VECTOR_PREDICATE_SHORT_CIRCUIT -DYY_NO_UNISTD_H
-D_ISOC11_SOURCE -I../ -I. -isystem ../third_party/protobuf/src -isystem
../third_party/googletest/googletest/include
-I../third_party/benchmark/include -Ithird_party/gflags/include
-I../third_party/glog/src -Ithird_party -I../third_party/re2
-I../third_party/tmb/include
-I../third_party/protobuf_cmake/../protobuf/src -std=c++14
-Wno-tautological-compare -Wall -pedantic -Werror -Wno-extended-offsetof
-march=native -Wno-return-type-c-linkage   -Wno-sign-compare -MMD -MT
parser/CMakeFiles/quickstep_parser_SqlLexer.dir/SqlLexer_gen.cpp.o -MF
parser/CMakeFiles/quickstep_parser_SqlLexer.dir/SqlLexer_gen.cpp.o.d -o
parser/CMakeFiles/quickstep_parser_SqlLexer.dir/SqlLexer_gen.cpp.o -c
parser/SqlLexer_gen.cpp
/home/marc/workspace/incubator-quickstep/build/parser/SqlLexer_gen.cpp:1370:84:
error: implicit conversion of NULL constant to 'bool'
[-Werror,-Wnull-conversion]
  if ( ! ( yyg->yy_buffer_stack ?
yyg->yy_buffer_stack[yyg->yy_buffer_stack_top] : __null) ) {
   ~
^~

   false
[... condensed because there were a total of 12]

/home/marc/workspace/incubator-quickstep/build/parser/SqlLexer_gen.cpp:3494:82:
error: implicit conversion of NULL constant to 'bool'
[-Werror,-Wnull-conversion]
 while(( yyg->yy_buffer_stack ?
yyg->yy_buffer_stack[yyg->yy_buffer_stack_top] : __null)){
 ~
  ^~

 false
12 errors generated.

The way I have been getting around this is to add set(CMAKE_CXX_FLAGS
"${CMAKE_CXX_FLAGS} -Wno-null-conversion") to the CMakeLists.txt

Anyone have a better way?

-Marc

Re: clang + ubuntu + BUILD_TYPE=RelWithDebugInfo

[Marc Spehlmann]

You're correct, setting the number of threads down to 2, I was able to
compile with clang. Thank you!

On Mon, Sep 5, 2016 at 11:08 AM, Harshad Deshmukh <hars...@cs.wisc.edu>
wrote:

> Hi Marc,
>
> On a CloudLab machine with Ubuntu 16, I did not face any issue with the
> RelWithDebInfo build type. That machine has a lot more memory though,
> compared to your laptop.
>
> One possibility: If you are running a multi-threaded make, the compilation
> requires more memory which triggers the behavior that you saw. In which
> case, can you try a single threaded build?
>
>
>
> On 09/05/2016 09:21 AM, Marc Spehlmann wrote:
>
>> Hey guys, has anyone experienced problems building on linux (Ubuntu 16) in
>> RelWithDebugInfo?
>>
>> I have no trouble building in debug mode with clang-3.8, clang-3.7, and
>> gcc
>> 5.4 but the compiler hangs up and eats all the memory on my laptop (8gb)
>> then dies (bottom of message) when in Rel. This happens when building our
>> templatized Comparitors.
>>
>> Any hints?
>> Marc
>>
>>
>> CLANG:
>> [754/1010] Building CXX object types/operations/comparisons/
>> CMakeFiles/quickstep_types_operations_comparisons_EqualComparison.dir/
>> EqualComparison.cpp.o
>> FAILED: types/operations/comparisons/CMakeFiles/quickstep_types_
>> operations_comparisons_EqualComparison.dir/EqualComparison.cpp.o
>> /usr/bin/clang++-3.7   -DQUICKSTEP_ENABLE_PROFILER
>> -DQUICKSTEP_ENABLE_VECTOR_COPY_ELISION_SELECTION
>> -DQUICKSTEP_ENABLE_VECTOR_PREDICATE_SHORT_CIRCUIT
>> -D_ISOC11_SOURCE -I../ -I. -isystem ../third_party/protobuf/src -isystem
>> ../third_party/googletest/googletest/include
>> -I../third_party/benchmark/include
>> -Ithird_party/gflags/include -I../third_party/glog/src -Ithird_party
>> -I../third_party/re2 -I../third_party/tmb/include
>> -I../third_party/protobuf_cmake/../protobuf/src
>> -std=c++14 -Wall -pedantic -Werror -Wno-extended-offsetof -march=native
>> -Wno-return-type-c-linkage -O2 -g -DNDEBUG -MMD -MT
>> types/operations/comparisons/CMakeFiles/quickstep_types_
>> operations_comparisons_EqualComparison.dir/EqualComparison.cpp.o -MF
>> types/operations/comparisons/CMakeFiles/quickstep_types_
>> operations_comparisons_EqualComparison.dir/EqualComparison.cpp.o.d -o
>> types/operations/comparisons/CMakeFiles/quickstep_types_
>> operations_comparisons_EqualComparison.dir/EqualComparison.cpp.o -c
>> ../types/operations/comparisons/EqualComparison.cpp
>> terminate called after throwing an instance of 'std::bad_alloc'
>>what():  std::bad_alloc
>> 0  libLLVM-3.7.so.1 0x7fa13bf4d050
>> llvm::sys::PrintStackTrace(llvm::raw_ostream&)
>> + 64
>> 1  libLLVM-3.7.so.1 0x7fa13bf4c5d1
>> 2  libpthread.so.0  0x7fa13a84f3d0
>> 3  libc.so.60x7fa139f12418 gsignal + 56
>> 4  libc.so.60x7fa139f1401a abort + 362
>> 5  libstdc++.so.6   0x7fa13a54b84d __gnu_cxx::__verbose_terminate
>> _handler()
>> + 365
>> 6  libstdc++.so.6   0x7fa13a5496b6
>> 7  libstdc++.so.6   0x7fa13a549701
>> 8  libstdc++.so.6   0x7fa13a549919
>> 9  libstdc++.so.6   0x7fa13a549ebc
>> 10 libLLVM-3.7.so.1 0x7fa13affaf88
>> 11 libLLVM-3.7.so.1 0x7fa13affb25a llvm::DILocation*
>> llvm::MDNode::storeImpl<llvm::DILocation, llvm::DenseSet> n*,
>> llvm::MDNodeInfo > >(llvm::DILocation*,
>> llvm::Metadata::StorageType, llvm::DenseSet<llvm::DILocation*,
>> llvm::MDNodeInfo >&) + 218
>> 12 libLLVM-3.7.so.1 0x7fa13affb39b
>> llvm::DILocation::getImpl(llvm::LLVMContext&,
>> unsigned int, unsigned int, llvm::Metadata*, llvm::Metadata*,
>> llvm::Metadata::StorageType, bool) + 283
>> 13 libLLVM-3.7.so.1 0x7fa13b004a26 llvm::DebugLoc::get(unsigned int,
>> unsigned int, llvm::MDNode const*, llvm::MDNode const*) + 38
>> 14 libLLVM-3.7.so.1 0x7fa13b42ec3d
>> 15 libLLVM-3.7.so.1 0x7fa13b43662d llvm::InlineFunction(llvm::Cal
>> lSite,
>> llvm::InlineFunctionInfo&, bool) + 2685
>> 16 libLLVM-3.7.so.1 0x7fa13b20da98
>> llvm::Inliner::runOnSCC(llvm::CallGraphSCC&)
>> + 3192
>> 17 libLLVM-3.7.so.1 0x7fa13b5deb7f
>> 18 libLLVM-3.7.so.1 0x7fa13b0b8d14
>> llvm::legacy::PassManagerImpl::run(llvm::Module&)
>> + 772
>> 19 clang0x00854c8a
>> clang::EmitBackendOutput(clang::DiagnosticsEngine&,
>> clang::CodeGenOptions const&, clang::TargetOptions const&,
>> clang::LangOptions const&, llvm::StringRef, llvm::Module*,
>> clang::BackendAction, llvm::raw_pwrite_stream*) + 3098
>&

vim -> IDE

[Marc Spehlmann]

Hello Quicksteppers,

If you use vim, I recommend trying the plugin YouCompleteMe (
https://github.com/Valloric/YouCompleteMe). It offers code completion and
it works well out of the box with quickstep. I'll give a quick intro on how
to get it set up:

1 Requirements
 - clang
 - vim 7.3.8+
 - ruby

2 Installation
 - I used Vundle (https://github.com/VundleVim/Vundle.vim), a plugin
manager for vim. There is a set up guide on the README. It was easy to get
started and I tried another couple plugins. One thing I noticed was that
sometimes vundle would not install correcly (with CommandT, for example) so
I had to manually install by going to ~/.vim/bundle/my-plugin and following
the README there.

My vimrc looks like:
```
" Vundle Config
set nocompatible  " be iMproved, required
filetype off  " required

" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()
" alternatively, pass a path where Vundle should install plugins
"call vundle#begin('~/some/path/here')

" let Vundle manage Vundle, required
Plugin 'VundleVim/Vundle.vim'
Plugin 'Valloric/YouCompleteMe'
Plugin 'wincent/command-t'
Plugin 'rdnetto/YCM-Generator'
" All of your Plugins must be added before the following line
call vundle#end()" required
filetype plugin indent on" required
" To ignore plugin indent changes, instead use:
"filetype plugin on
"
" Brief help
" :PluginList   - lists configured plugins
" :PluginInstall- installs plugins; append `!` to update or just
:PluginUpdate
" :PluginSearch foo - searches for foo; append `!` to refresh local cache
" :PluginClean  - confirms removal of unused plugins; append `!` to
auto-approve removal
"
" see :h vundle for more details or wiki for FAQ

" My config
```
 - Then I modified quickstep's CMakeList.txt, adding a line to create a
compilation commands database:
 `set( CMAKE_EXPORT_COMPILE_COMMANDS 1 )`
 -Recompile quickstep, it will generate a json file in the build folder
with the database
 -Add a custom YCM config file to the root quickstep directory. Here's the
one I used (note the !!! CHANGE THIS comments)

 ```
import os
import os.path
import fnmatch
import logging
import ycm_core

# !!! CHANGE THIS
# I used this before I found out about the flags database. If you
# generate it, you will not need any flags:
flags = [
'-DQUICKSTEP_DEBUG',
'-DQUICKSTEP_ENABLE_VECTOR_COPY_ELISION_SELECTION',
'-DQUICKSTEP_ENABLE_VECTOR_PREDICATE_SHORT_CIRCUIT',
'-DYY_NO_UNISTD_H',
'-D_ISOC11_SOURCE',
'-I',
'/Users/mspehlmann/code/incubator-quickstep',
'-I',
'/Users/mspehlmann/code/incubator-quickstep/build',
'-isystem',
'/Users/mspehlmann/code/incubator-quickstep/third_party/protobuf/src',
'-isystem',

'/Users/mspehlmann/code/incubator-quickstep/third_party/googletest/googletest/include',
'-I',

'/Users/mspehlmann/code/incubator-quickstep/third_party/benchmark/include',
'-I',

'/Users/mspehlmann/code/incubator-quickstep/build/third_party/gflags/include',
'-I',
'/Users/mspehlmann/code/incubator-quickstep/third_party/glog/src',
'-I',
'/Users/mspehlmann/code/incubator-quickstep/build/third_party',
'-I',
'/Users/mspehlmann/code/incubator-quickstep/third_party/re2',
'-I',
'/Users/mspehlmann/code/incubator-quickstep/third_party/tmb/include ',
'-std=c++14',
'-Wall',
'-pedantic',
'-Werror',
'-march=native',
'-Wno-return-type-c-linkage',
'-g'
]

SOURCE_EXTENSIONS = [
'.cpp',
'.cxx',
'.cc',
'.c',
'.m',
'.mm'
]

HEADER_EXTENSIONS = [
'.h',
'.hxx',
'.hpp',
'.hh'
]
# !!! CHANGE THIS
compilation_database_folder =
'/Users/mspehlmann/code/incubator-quickstep/build'

if os.path.exists( compilation_database_folder ):
  database = ycm_core.CompilationDatabase( compilation_database_folder )
else:
  database = None

SOURCE_EXTENSIONS = [ '.cpp', '.cxx', '.cc', '.c', '.m', '.mm' ]

def DirectoryOfThisScript():
  return os.path.dirname( os.path.abspath( __file__ ) )


def MakeRelativePathsInFlagsAbsolute( flags, working_directory ):
  if not working_directory:
return list( flags )
  new_flags = []
  make_next_absolute = False
  path_flags = [ '-isystem', '-I', '-iquote', '--sysroot=' ]
  for flag in flags:
new_flag = flag

if make_next_absolute:
  make_next_absolute = False
  if not flag.startswith( '/' ):
new_flag = os.path.join( working_directory, flag )

for path_flag in path_flags:
  if flag == path_flag:
make_next_absolute = True
break

  if flag.startswith( path_flag ):
path = flag[ len( path_flag ): ]
new_flag = path_flag + os.path.join( working_directory, path )
break

if new_flag:
  new_flags.append( new_flag )
  return new_flags


def IsHeaderFile( filename ):
  extension = os.path.splitext( filename )[ 1 ]
  return extension in [ '.h', '.hxx', '.hpp', '.hh' ]


def