Hi Michalis, Sure! It was more of a heads-up than anything else. Please don’t hesitate to pose your questions here or to call for a meeting if necessary :)
Best, Jorge > On 27 Apr 2023, at 16.52, Michalis Vargiamis <[email protected]> > wrote: > > Hi! > > > Thanks! Actually I've lost quite some time re configuring my setup. I > currently have a windows laptop, so for the Linux I initially tried the WSL > that windows provide but it turned out to make the whole > development/debugging process quite inefficient, so i switched to vmware but > then again I lost some time with some other stuff. Anyway, I'd have to ask > for a bit more time on this one. > > > Thank you, > > Michalis > > > On 27-Apr-23 10:41 AM, Jorge Arnulfo Quiané Ruiz wrote: >> Hi Michalis, >> >> Please let us know if you need help :) >> >> Best, >> Jorge >> >>> On 25 Apr 2023, at 14.07, Zoi Kaoudi <[email protected]> wrote: >>> >>> Hi Michalis, >>> can you double check that you define the types of the Tuple2 output? >>> >>> >>> According to the error >>> "Return type >>> PojoType<org.apache.wayang.basic.data.Tuple2, fields = [field0: >>> GenericType<java.lang.Object>, field1: GenericType<java.lang.Object>]>"it >>> seems like the Tuple2 does not have specific types but they are >>> java.lang.Objects. Maybe that could be the problem. >>> >>> Also to give some context, if you see in the >>> FlinkMaterializedGroupByOperator code there are some utility functions we >>> use to map the Wayang UDFs to the Flink (or Spark) UDFs. For example, the >>> line: >>> final KeySelector<Type, KeyType> keyExtractor = >>> flinkExecutor.getCompiler().compileKeySelector(this.getKeyDescriptor()); >>> converts the Wayang UDF keyDescriptor to a Flink UDF KeySelector. Maybe if >>> you check the code of this class you could spot the problem? >>> Best >>> -- >>> Zoi >>> >>> Στις Τρίτη 25 Απριλίου 2023 στις 01:30:20 μ.μ. CEST, ο χρήστης Michalis >>> Vargiamis <[email protected]> έγραψε: >>> >>> Hello! >>> >>> >>> I've been working with the missing operator tests for Flink. I've >>> successfully done the SortOperator and the UnionAllOperator by seeing >>> the respective Spark operator tests and modifying RddChannel to >>> DataSetChannel. >>> >>> >>> I'm having trouble with the tests for other operators though, for >>> example the FlinkMaterializedGroupByOperator. I tried starting with >>> SparkMaterializedGroupByOperatorTest and doing the same RddChannel to >>> DataSetChannel modifications as before, but I get the following error: >>> >>> >>> [ERROR] >>> org.apache.wayang.flink.operators.FlinkMaterializedGroupByOperatorTest.testExecution >>> Time elapsed: 1.911 s <<< ERROR! >>> org.apache.flink.api.common.InvalidProgramException: Return type >>> PojoType<org.apache.wayang.basic.data.Tuple2, fields = [field0: >>> GenericType<java.lang.Object>, field1: GenericType<java.lang.Object>]> >>> of KeySelector class >>> org.apache.wayang.flink.compiler.KeySelectorFunction is not a valid key type >>> at >>> org.apache.wayang.flink.operators.FlinkMaterializedGroupByOperatorTest.testExecution(FlinkMaterializedGroupByOperatorTest.java:50) >>> >>> >>> Digging into the operator code a bit more, the error happens at >>> >>> dataSetInput.groupBy(keyExtractor); >>> >>> >>> Any ideas on what should be changed? >>> >>> >>> Here is a permalink to the respective spark test >>> [https://github.com/apache/incubator-wayang/blob/6aad4eea8c91a52f2a41e79424491e6c2c5206af/wayang-platforms/wayang-spark/code/test/java/org/apache/wayang/spark/operators/SparkMaterializedGroupByOperatorTest.java] >>> >>> >>> Thank you, >>> >>> Michalis Vargiamis >>>
