Re: Article: The Rise of the Data Engineer

2017-01-25 Thread Gerard Toonstra
> > > > Per this email thread, it almost sounds like a slack team/discourse for > data engineering might be useful. > > I certainly would not mind getting more knowledge on this topic and I'd like to be invited to such a slack group (or google group).

Re: Article: The Rise of the Data Engineer

2017-01-25 Thread Brian Van Klaveren
There's also monetdb and greenplum, depending on your data size, which https://www.monetdb.org, which support columnar tables if you want to get your feet wet. If your data is actually more array-like, you might try out scidb. Per this email thread, it almost sounds like a slack team/discourse f

Re: Article: The Rise of the Data Engineer

2017-01-25 Thread Rob Goretsky
@Gerard - I mentioned Vertica just as one of the first examples of a system that offers columnar storage. You might actually see a significant benefit using columnar storage with even a smaller table, as small as a few GB - Columnar storage works well if you have wide fact tables with many columns

Re: Article: The Rise of the Data Engineer

2017-01-25 Thread Boris Tyukin
Max, really really nice post and I like your style of writing - please continue sharing your experience and inspire many of us working in more traditional environments ;) I shared your post with our leadership and hopefully we will have data engineers soon on our team! As far as UI vs. coding, I am

Re: Article: The Rise of the Data Engineer

2017-01-25 Thread Gerard Toonstra
You mentioned Vertica and Parquet. Is it recommended to use these newer tools even when the DWH is not BigData size (about 150G in size) ? So there are a couple of good benefits, but are there any downsides and disadvantages you have to take into account comparing Vertica vs. SQL Server for exampl

Re: Article: The Rise of the Data Engineer

2017-01-24 Thread Maxime Beauchemin
Glad to hear the article resonated with you! I just now got interviewed on a podcast on this very subject, it should be up sometime this week: https://itunes.apple.com/us/podcast/data-engineering-podcast/id1193040557 It's less structured than the article, but you can hear me babble about data engi

Re: Article: The Rise of the Data Engineer

2017-01-24 Thread Rob Goretsky
Maxime, Just wanted to thank you for writing this article - much like the original articles by Jeff Hammerbacher and DJ Patil coining the term "Data Scientist", I feel this article stands as a great explanation of what the title of "Data Engineer" means today.. As someone who has been working in t

Re: Article: The Rise of the Data Engineer

2017-01-23 Thread Laura Lorenz
👍 On Fri, Jan 20, 2017 at 7:38 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hey I just published an article about the "Data Engineer" role in modern > organizations and thought it could be of interest to this community. > > https://medium.com/@maximebeauchemin/the-rise-of-the-data

Article: The Rise of the Data Engineer

2017-01-20 Thread Maxime Beauchemin
Hey I just published an article about the "Data Engineer" role in modern organizations and thought it could be of interest to this community. https://medium.com/@maximebeauchemin/the-rise-of-the-data-engineer-91be18f1e603#.5rkm4htnf Max