Re: Drill TupleMetadata created from DFDL Schema - how do I inform Drill about it

Hi Mike, I hope all is well. I remembered one other piece which might be useful for you. Drill has an interface called a PersistentStore which is used for storing artifacts such as tokens etc. I've uesd it on two occasions: in the GoogleSheets plugin and the Http plugin. In both cases, I us

Re: Drill TupleMetadata created from DFDL Schema - how do I inform Drill about it

Hi Charles, The persistent store is just ZooKeeper, and ZK is known to work poorly as a distributed DB. ZK works great for things like tokens, node registrations and the like. But, ZK scales very poorly for things like schemas (or query profiles or a list of active queries.) A more scalable appro

Re: Drill TupleMetadata created from DFDL Schema - how do I inform Drill about it

I am very much hoping someone will look at my open PR soon. https://github.com/apache/drill/pull/2836 I am basically blocked on this effort until you help me with one key area of that. I expect the part I am puzzling over is routine to you, so it will save me much effort. This is the key area in

Re: Drill TupleMetadata created from DFDL Schema - how do I inform Drill about it

Got it. I’ll review today and tomorrow and hopefully we can get you unblocked. Sent from my iPhone > On Oct 18, 2023, at 18:01, Mike Beckerle wrote: > > I am very much hoping someone will look at my open PR soon. > https://github.com/apache/drill/pull/2836 > > I am basically blocked on thi

Re: Drill TupleMetadata created from DFDL Schema - how do I inform Drill about it

Hi Mike, Earlier on, there were two approaches discussed: 1. Using a Daffodil schema to map to a Drill schema, and use Drill's existing schema mechanisms for all of Drill's existing input formats. 2. Using a Daffodil-specific reader so that Daffodil does the data parsing. Some of my earlier answ

Re: [PR] WIP: Preliminary Review on adding Daffodil to Drill (drill)

cgivre commented on code in PR #2836: URL: https://github.com/apache/drill/pull/2836#discussion_r1364795241 ## contrib/format-daffodil/src/main/java/org/apache/drill/exec/store/daffodil/DaffodilBatchReader.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] WIP: Preliminary Review on adding Daffodil to Drill (drill)

cgivre commented on code in PR #2836: URL: https://github.com/apache/drill/pull/2836#discussion_r1364797516 ## contrib/format-daffodil/src/test/java/org/apache/drill/exec/store/daffodil/TestDaffodilReader.java: ## @@ -0,0 +1,652 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] WIP: Preliminary Review on adding Daffodil to Drill (drill)

cgivre commented on code in PR #2836: URL: https://github.com/apache/drill/pull/2836#discussion_r1364798270 ## distribution/src/assemble/component.xml: ## Review Comment: Please keep these in alphabetical order. -- This is an automated message from the Apache Git Servic

Re: [PR] WIP: Preliminary Review on adding Daffodil to Drill (drill)

cgivre commented on code in PR #2836: URL: https://github.com/apache/drill/pull/2836#discussion_r1364797899 ## contrib/format-daffodil/src/test/java/org/apache/drill/exec/store/daffodil/TestDaffodilReader.java: ## @@ -0,0 +1,652 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] WIP: Preliminary Review on adding Daffodil to Drill (drill)

cgivre commented on code in PR #2836: URL: https://github.com/apache/drill/pull/2836#discussion_r1364798604 ## contrib/pom.xml: ## @@ -59,6 +59,7 @@ format-pcapng format-iceberg format-deltalake +format-daffodil Review Comment: Please keep these in alphabe