---------- Forwarded message --------- From: Igor Guzenko <ihor.huzenko....@gmail.com> Date: Fri, Jan 10, 2020 at 1:46 PM Subject: Re: About integration of drill and arrow To: dev <d...@drill.apache.org>
Hello Drill Developers and Drill Users, This discussion started as migration to Arrow but uncovered questions of strategical plans for moving towards Apache Drill 2.0. Below are my personal thoughts of what we, as developers, should do to offer Drill users better experience: 1. High performant bulk insertions into as many data sources as possible. There is a whole bunch of different tools for data pipelining to use... But why people who know SQL should spend time learning something new for simply moving data between tools? 2. Improve the efficiency of memory management (EVF, resource management, improved costs planning using meta store, etc.). Since we're dealing with big data alongside other tools installed on data nodes we should utilize memory very economically and effectively. 3. Make integration with all other tools and formats as stable as possible. The high amount of bugs in the area tells that we have lots to improve. Every user is happy when he gets a tool and it simply works as expected. Also, analyze user requirements and provide integration with new most popular tools. Querying high variety of data sources were and still one of the biggest selling points. 4. Make code highly extensible and extremely friendly for contributions. No one would want to spend years of learning to make a contribution. This is why I want to see a lot of modules that are highly cohesive and define clear APIs for interaction with each other. This is also about paying old technical debts related to fat JDBC client, copy of web server in Drill on YARN, mixing everything in exec module, etc. 5. Focus on performance improvements of every component, from query planning to execution. These are my thoughts from developer's perspective. Since I'm just developer from Ukraine and far far away from Drill users, I believe that Charles Givre is the one who can build a strong Drill user community and collect their requirements for us. What relates to Volodymyr's suggestion about adapting Arrow and Drill vectors to work together (the same step is required to implement an Arrow client, suggested by Paul). I'm totally against the idea because it brings a huge amount of unnecessary complexity just to uncover small insides into the integration. First is that this is against the whole idea of Arrow since the main idea of Arrow is to provide unified columnar memory layout between different tools without any data conversions. But the step exactly requires data conversions, at least our nullability vector and their validity bitmaps are not the same, also Dict vector and their meaning of Dict may also cause data conversion. Another waste is the difference in metadata contracts, who knows whether it's even possible to combine them. Another problem, like I already mentioned is the huge complexity of the work, To do the work I should overcome all underlying pitfalls of both projects, in addition, I should cover all the untestable code with a comprehensive amount of tests to show that back and forth conversion is done correctly for every single unit of data in both vectors. The idea of adapters and clients is about 4 years old or more and no one did practical work to implement it. I think I explained why. What I really like in Volodymyr's and Paul's suggestions is that we can extract clear API from existing EVF implementation and in practice provide Arrow or any other implementation for it. Who knows, maybe with new improved garbage collectors using direct memory is not necessary at all? It is quite clear what we need the middle layer between operators and memory, we need extensive benchmarks over the layer and experiments to show what is the best underlying memory for Drill. What about client tools compatibility there is only one solution I can see is to provide new clients for Drill 2.0, although I agree that this is a tremendous amount of work there is no other way for making major steps into the future. Without it, we should lay back and watch while Drill is slowly dying and giving up to its competitors. NOTE: I want to encourage everyone to join the discussion and share vision of what should be included in Drill 2.0 and what are strategic points we want to achieve in the future. Kind regards, Igor On Thu, Jan 9, 2020 at 10:12 PM Paul Rogers <par0...@yahoo.com.invalid> wrote: > Hi Volodymyr, > > All good points. The Arrow/Drill conversion is a good option, especially > for readers and clients. Between operators, such conversion is likely to > introduce performance hits. As you know, the main feature that > differentiates one query engine from another is performance, so adding > conversions is unlikely to help Drill in the performance battle. > > Flatten should actually be pretty simple with EVF. Creating repeated > values is much like filling in implicit columns: set a value, then "copy > down" n times. > > Still, you raise good issues. Operators that fit your description are > things like exchanges: these operators want to work at the low level of > buffers. Not much the column readers/writers can do to help. And, as you > point out, commercial components will be a challenge as Apache Drill does > not maintain that code. > > Your larger point is valid: no matter how we approach it, moving to Arrow > is a large project that will break compatibility. > > We've discussed a simple first step: Support an Arrow client to see if > there is any interest. Support Arrow readers to see if that gives us any > benefit. These are the visible tip of the iceberg. If we see advantages, we > can then think about changing the internals; the vast bulk of the iceberg > which is below water and unseen. > > I think I disagree that we'd want to swap code that works directly with > ValueVectors to code that works directly with ArrowVectors. Doing so locks > us into someone else's memory format and forces Drill to change every time > Arrow changes. Give the small size of the Drill team, and the frantic pace > of Arrow change, This was the team's concern early on and I'm still not > convinced this is a good strategy. > > So, my original point remains: what is the benefit of all this cost? Is > there another path that gives us greater benefit for the same or lesser > cost? In short, what is our goal? Where do we want Drill to go? > > In fact, another radical suggestion is to embrace the wonderful work done > on Presto. Maybe Drill 2.0 is simply Presto. We focus on adding support for > local files (Drill's unique strength), and embrace Presto's great support > for data types, connectors, UDFs, clients and so on. > > As a team, we should ask the fundamental question: What benefits can Drill > offer that are not already offered by, say Presto or commercial Drill > derivatives? If new can answer that question, we'll have a better idea > about whether investment in Arrow will get us there. > > Or, are we better off to just leave well enough alone as we have done for > several years? > > Thanks, > - Paul > > > > On Thursday, January 9, 2020, 05:57:52 AM PST, Volodymyr Vysotskyi < > volody...@apache.org> wrote: > > Hi all, > > Glad to see that this discussion became active again! > > I have some comments regarding the steps for moving from Drill Vectors to > Arrow Vectors. > > No doubt that using EVF for all operators and readers instead of value > vectors will simplify things a lot. > But considering the target goal - integration with Arrow, it may be the > main show-stopper for it. > There may be some operators which would be hard to adapt to use EVF, for > example, I think Flatten operator will be among them since its > implementation deeply connected with value vectors. > Also, it requires moving all storage and format plugins to EVF, which also > may be problematic, for example, some plugins like MaprDB have specific > features, and it should be considered when moving to EVF. > Some other plugins are so obsolete, that I'm not sure that they still work > and that someone still uses it, so except moving to EVF, they should be > resurrected to verify that they weren't broken more than before. > > This is a huge piece of work, and only after that, we will proceed with the > next step - integrating Arrow to EVF and then handling new Arrow-related > issues for all the operators and readers at the same time. > > I propose to update these steps a little bit. > 1. I agree that at first, we should extract EVF-related classes into a > separate module. > 2. But as the next step, I propose to extract EVF API which doesn't depend > on the vector implementation (Drill vectors, or Arrow ones). > 3. After that, introduce module with Arrow which also implements this EVF > API. > 4. Introduce transformers that will be able to convert from Drill vectors > into Arrow vectors and vice versa. These transformers may be implemented to > work using EVF abstractions instead of operating with specific vector > implementations. > > 5.1. At this point, we can introduce Arrow connectors to fetch the data in > Arrow format or return it in such a format using transformers from step 4. > > 5.2. Also, at this point, we may start rewriting operators to EVF and > switching EVF implementation from the EVF based on Drill Vectors to the > implementation which uses Arrow Vectors. Or switching implementations for > existing EVF-based format plugins and fix newly discovered issues in Arrow. > Since at this point we will have operators which use Arrow format and > operators which use Drill Vectors format, we should insert operators that > transform one vector format to another introduced in step 4 between every > pair of operators which returns batches in a different format. > > I know, that such an approach requires some additional work, like > introducing transformers from step 4 and may cause some performance > degradations for the case when format transformation is complex for some > types and when we still have sequences of operators with different formats. > > But with this approach, transitioning to Arrow wouldn't be blocked until > everything is moved to EVF and it would be possible to transmit > step-by-step, and Drill still will be able to switch between formats if it > would be required. > > Kind regards, > Volodymyr Vysotskyi > > > On Thu, Jan 9, 2020 at 2:45 PM Igor Guzenko <ihor.huzenko....@gmail.com> > wrote: > > > Hi Paul, > > > > Though I have very limited knowledge about Arrow at the moment, I can > > highlight a few advantages of trying it: > > 1. Allows fixing all the long-standing nullability issues and provide > > better integration for storage plugins like Hive. > > https://jira.apache.org/jira/browse/DRILL-1344 > > https://jira.apache.org/jira/browse/DRILL-3831 > > https://jira.apache.org/jira/browse/DRILL-4824 > > https://jira.apache.org/jira/browse/DRILL-7255 > > https://jira.apache.org/jira/browse/DRILL-7366 > > 2. Some work was done by community to implement optimized Arrow readers > for > > Parquet and other formats&tools. We could try to adopt and check whether > we > > can benefit from them. > > 3. Since Arrow is under active development we could try their newest > > features, like Flight which promises improved data transfers over the > > network. > > > > Thanks, > > Igor > > On Wed, Jan 8, 2020 at 11:55 PM Paul Rogers <par0...@yahoo.com.invalid> > > wrote: > > > > > Hi Igor, > > > > > > Before diving into design issues, it may be worthwhile to think about > the > > > premise: should Drill adopt Arrow as its internal memory layout? This > is > > > the question that the team has wrestled with since Arrow was launched. > > > Arrow has three parts. Let's think about each. > > > > > > First is a direct memory layout. The approach you suggest will let us > > work > > > with the Arrow memory format. Use EVF to access vectors, then the > > > underlying vectors can be swapped from Drill to Arrow. But, what is the > > > advantage of using Arrow? The arrow layout isn't better than Drill's; > it > > is > > > just different. Adopting the Arrow memory layout by itself provides > > little > > > benefit, but bit cost. This is one reason the team has been so > reluctant > > to > > > atop Arrow. > > > > > > The only advantage of using the Arrow memory layout is if Drill could > > > benefit from code written for Arrow. The second part of Arrow is a set > of > > > modules to manipulate vectors. Gandiva is the most prominent example. > > > However, there are major challenges. Most SQL operations are defined to > > > work on rows; some clever thinking will be needed to convert those > > > operations into a series of column operations. (Drill's codegen is NOT > > > columnar: it works row-by-row.) So, if we want to benefit from Gandiva, > > we > > > must completely rethink how we process batches. > > > > > > Is it worth doing all that work? The primary benefit would be > > performance. > > > But, it is not clear that our current implementation is the bottleneck. > > The > > > current implementation is row-based, code generated in Java. Would be > > great > > > for someone to do some benchmarks to show the benefit from adopting > > Gandiva > > > to see if the potential gain justifies the likely large development > cost. > > > > > > The third advantage of using Arrow is to allow exchange of vectors > > between > > > Drill and Arrow-based clients or readers. As it turns out, this is not > > the > > > big win it seems. As we've discussed, we could easily create an > > Arrow-based > > > client for Drill -- there will be an RPC between the client and Drill > and > > > we can use that to do format conversion. > > > > > > For readers, Drill will want control over batch sizes; Drill cannot > > > blindly accept whatever size vectors a reader chooses to produce. (More > > on > > > that later.) Incoming data will be subject to projection and selection, > > so > > > it will quickly move out of the incoming Arrow vectors into vector > which > > > Drill creates. > > > > > > Arrow gets (or got) a lot of press. However, our job is to focus on > > what's > > > best for Drill. There actually might be a memory layout for Drill that > is > > > better than Arrow (and better than our current vectors.) A couple of us > > did > > > a prototype some time ago that seemed to show promise. So, it is not > > clear > > > that adopting Arrow is necessarily a huge win: maybe it is, maybe not. > We > > > need to figure it out. > > > > > > What IS clearly a huge win is the idea you outlined: creating a layer > > > between memory layout and the rest of Drill so that we can try out > > > different memory layouts to see what works best. > > > > > > Thanks, > > > - Paul > > > > > > > > > > > > On Wednesday, January 8, 2020, 10:02:43 AM PST, Igor Guzenko < > > > ihor.huzenko....@gmail.com> wrote: > > > > > > Hello Paul, > > > > > > I totally agree that integrating Arrow by simply replacing Vectors > usage > > > everywhere will cause a disaster. > > > After the first look at the new *E*nhanced*V*ector*F*ramework and based > > on > > > your suggestions I think I have an idea to share. > > > In my opinion, the integration can be done in the two major stages: > > > > > > *1. Preparation Stage* > > > 1.1 Extract all EVF and related components to a separate module. > So > > > the new separate module will depend only upon Vectors module. > > > 1.2 Step-by-step rewriting of all operators to use a higher-level > > > EVF module and remove Vectors module from exec and modules > dependencies. > > > 1.3 Ensure that only module which depends on Vectors is the new > EVF > > > one. > > > *2. Integration Stage* > > > 2.1 Add dependency on Arrow Vectors module into EVF module. > > > 2.2 Replace all usages of Drill Vectors & Protobuf Meta with > > Arrow > > > Vectors & Flatbuffers Meta in EVF module. > > > 2.3 Finalize integration by removing Drill Vectors module > > > completely. > > > > > > > > > *NOTE:* I think that any way we won't preserve any backward > compatibility > > > for drivers and custom UDFs. > > > And proposed changes are a major step forward to be included in Drill > 2.0 > > > version. > > > > > > > > > Below is the very first list of packages that in future may be > > transformed > > > into EVF module: > > > *Module:* exec/Vectors > > > *Packages:* > > > org.apache.drill.exec.record.metadata - (An enhanced set of classes to > > > describe a Drill schema.) > > > org.apache.drill.exec.record.metadata.schema.parser > > > > > > org.apache.drill.exec.vector.accessor - (JSON-like readers and writers > > for > > > each kind of Drill vector.) > > > org.apache.drill.exec.vector.accessor.convert > > > org.apache.drill.exec.vector.accessor.impl > > > org.apache.drill.exec.vector.accessor.reader > > > org.apache.drill.exec.vector.accessor.writer > > > org.apache.drill.exec.vector.accessor.writer.dummy > > > > > > *Module:* exec/Java Execution Engine > > > *Packages:* > > > org.apache.drill.exec.physical.rowSet - (Record batches management) > > > org.apache.drill.exec.physical.resultSet - (Enhanced rowSet with memory > > > mgmt) > > > org.apache.drill.exec.physical.impl.scan - (Row set based scan) > > > > > > Thanks, > > > Igor Guzenko > > > > > > > > >
