xiarixiaoyao edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-954487215
@vinothchandar thanks for your suggestion. the question:Side note: At-least from IDE builds, I get the following error when running tests. did you fae this? have you tried running this on a cluster using a spark bundle? ans: i use java 8 to run those code , of course i test those code in cluster. but i find if we uses java 11 to run those code in IDE, IDE will throw "package sun.misc. does not exist", since Packages com.sun.* and sun.* hold internal stuff, and should not be used by thirdparty applications (like yours probably) in general case. now i removed all unsafe function, it should be ok for java 11. update the code. 1. rewrite most of scala functions(ZOptimze.scala) by java in ZCurveOptimizeHelper.java and remove ZOptimze.scala 2. remove unsafe lexicographical cmp and use java lexicographical cmp. now no unsafe class exisit. 3. dataskipping should work with sql and datasource, not just datasource. 4. use a config to control dataskipping in HoodieFileIndex, defensively protect the existing code path. i think we may no need to move data skipping config to DataSourceReadOptions. we can use it in HoodieFIleIndex directly to control if enable dataskipping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
