Do you also need to add forceBits to the template FixedValueVectors.java ? Can you copy the line from VarLengthColumnReaders .java where you hit a compile error ?
On Fri, Feb 5, 2016 at 9:26 AM, Dave Oshinsky <[email protected]> wrote: > Hi Abdel, > Thanks for reply. I added it to > exec/vector/target/codegen/templates/NullableValueVectors.java. I then see > it propagated by Freemarker to generated sources, as shown below > signature. In Eclipse, I'm able to navigate from where I'm calling > forceBits() to e.g. NullableDecimal28SparseVector.java, and I see > forceBits() implemented there. However, Eclipse has the dreaded "red X" in > VarLengthColumnReaders.java indicating an error, and the build also fails. > Weird, huh? > > I do have to say, this whole "generated sources" business is a pain in the > you know where ;-) In all seriousness, I would implement the whole > decimal business MUCH more simply if writing it from scratch. Why the > gazillion special cases for different widths (all with separate generated > sources), for nullable and not, sparse or dense? Just treat them all the > same, at least from where I'm sitting with a Parquet-centric perspective. > And the memory usage would be MUCH lower by treating decimals ALWAYS as > variable width. Most actual decimal numbers are way smaller than the > precision would indicate, in many of my test cases. > > Dave Oshinsky > > $ find . -name "*.java" | xargs grep forceBits > ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: > nullableDecimal28Vector.forceBits(index, 1); > ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: > nullableDecimal38Vector.forceBits(index, 1); > ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: > nullableDecimal28Vector.forceBits(index, 1); > ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: > nullableDecimal38Vector.forceBits(index, 1); > ./exec/vector/src/main/codegen/templates/NullableValueVectors.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/classes/codegen/templates/NullableValueVectors.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/codegen/templates/NullableValueVectors.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBigIntVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBitVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDateVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal18Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28DenseVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28SparseVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38DenseVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38SparseVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal9Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat4Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat8Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalDayVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalYearVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableSmallIntVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeStampVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTinyIntVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt1Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt2Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt4Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt8Vector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVar16CharVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarBinaryVector.java: > public void forceBits(int index, int value){ // DAO > ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarCharVector.java: > public void forceBits(int index, int value){ // DAO > > -----Original Message----- > From: Abdel Hakim Deneche [mailto:[email protected]] > Sent: Friday, February 05, 2016 12:10 PM > To: [email protected] > Subject: Re: strange build problem with generated sources > > Hey Dave, > > Which file did you add the "forceBits()" method to ? > > On Fri, Feb 5, 2016 at 8:33 AM, Dave Oshinsky <[email protected]> > wrote: > > > Hi Drill-ers, > > I am looking into fixing JIRA > > https://issues.apache.org/jira/browse/DRILL-4184. I've encountered a > > number of strange build problems along the way with my drill 1.4 > > snapshot, including inability to rebuild after running "mvn clean", no > > matter what I try. So, I'm building from scratch for the second time, > > at least. The latest problem really has me stumped at the moment. I > > added a "forceBits(int,int)" method that I see in generated source > > file NullableDecimal28SparseVector.java (and > > NullableDecimal38SparseVector.java), but somehow this doesn't get > > compiled properly into the *.class and my build keeps failing as if > > the new forceBits method isn't there: > > > > [ERROR] Failed to execute goal > > org.apache.maven.plugins:maven-compiler-plugin:3.2:compile > > (default-compile) on project drill-java-exec: Compilation failure: > > Compilation failure: > > [ERROR] > > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m > > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo > > lumnReaders.java:[108,29] > > error: cannot find symbol > > [ERROR] symbol: method forceBits(int,int) > > [ERROR] location: variable nullableDecimal28Vector of type > > NullableDecimal28SparseVector [ERROR] > > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m > > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo > > lumnReaders.java:[179,29] > > error: cannot find symbol > > > > Can anyone suggest how to fix this without starting over from scratch > > in a new build node (again)? Any advice would be greatly appreciated. > > > > I will send a separate email eventually regarding the design of my > > fix, which I know is only a short-term solution to the problem of > > handling variable width decimal fields in Parquet files. To make a > > long story short, all the decimal vectors are fixed width vectors, > > which don't have the ability to "remember" varying sizes from one > > decimal field to the next. I've hacked up something to "remember" the > > varying field sizes (BigDecimal array sizes) in > > NullableVarLengthValuesColumn and VarLengthValuesColumn, not in the > > decimal vectors. This seems to work, though it's admittedly ugly. > > However, I ran into a problem with nullable varying width decimal > > columns where the "isSet" always returns 0, as if the column is null, > > when it is not, and the sparse decimal data is present in the vector > > (but Drill won't send the decimal value, because it thinks it's null). > > Hence the "forceBits" hack to try to work around this. It seemed like > > I was close to running a successful Drill query on the varying width > > decimal Parquet data, but alas, I ran into (another) build problem. I > > do have a LOT of questions as to why the decimal stuff was designed the > way it is, but that's for another email.... > > > > Thanks, > > Dave Oshinsky > > > > > > > > > > > > ***************************Legal Disclaimer*************************** > > "This communication may contain confidential and privileged material > > for the sole use of the intended recipient. Any unauthorized review, > > use or distribution by others is strictly prohibited. If you have > > received the message by mistake, please advise the sender by reply > > email and delete the message. Thank you." > > ********************************************************************** > > > > > -- > > Abdelhakim Deneche > > Software Engineer > > <http://www.mapr.com/> > > > Now Available - Free Hadoop On-Demand Training < > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > ***************************Legal Disclaimer*************************** > "This communication may contain confidential and privileged material for > the > sole use of the intended recipient. Any unauthorized review, use or > distribution > by others is strictly prohibited. If you have received the message by > mistake, > please advise the sender by reply email and delete the message. Thank you." > ********************************************************************** > -- Abdelhakim Deneche Software Engineer <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
