+1 On Mon, Nov 27, 2017 at 6:18 PM, Ruslan Dautkhanov <[email protected]> wrote:
> ‘’’ > I'm > now working on sql scanners, extractors and other tools that allow me to > populate the database > ‘’’ > > Very cool. Cloudera Navigator ( not an open source product) does this too > to some extent - collect metadata and create data lineage automatically ( > stored as a Solr collection) by parsing sql queries. > > https://www.cloudera.com/documentation/enterprise/5-12- > x/topics/datamgmt_extraction_indexing.html > > > > On Mon, Nov 27, 2017 at 12:38 PM Gerard Toonstra <[email protected]> > wrote: > > > Hi all, > > > > So something that really drew my attention recently is a "data portal" > as > > described by a team from airbnb somewhere in May. The idea is basically a > > "facebook of data": > > > > > > > > https://medium.com/airbnb-engineering/democratizing- > data-at-airbnb-852d76c51770 > > > > > > Unfortunately it looks like it's not going to be opensourced due to how > > heavily integrated it is with their specific infrastructure; but the idea > > itself to me sounds like it's something every organization of a certain > > size should have to keep track of data and stay informed as an > > organization. > > > > Based on the descriptions, I prototyped some things away and am happy > with > > the results and the speed that something like this can be constructed. > I'm > > now working on sql scanners, extractors and other tools that allow me to > > populate the database and put a poc together on some real data. > > > > If other people have similar concerns in their organization and think > this > > would be a great thing to have, reply to me or the list; with sufficient > > interest I may set up a web chat/meet session so this can be discussed in > > more detail and find ways to progress this. > > > > > > Best regards, > > > > Gerard > > >
