Thanks for these proposals. good suggestions. Is this style of breaking
down your approach standard?

My view would be that perhaps it makes more sense to follow the industry
established approach of breaking down your technical proposal  into:

   1. Background
   2. Objective
   3. Scope
   4. Constraints
   5. Assumptions
   6. Reporting
   7. Deliverables
   8. Timelines
   9. Appendix

Your current approach using below

Q1. What are you trying to do? Articulate your objectives using absolutely
no jargon. What are you trying to achieve?

Q2. What problem is this proposal NOT designed to solve? What issues the
suggested proposal is not going to address

Q3. How is it done today, and what are the limits of current practice?

Q4. What is new in your approach approach and why do you think it will be
successful succeed?

Q5. Who cares? If you are successful, what difference will it make? If your
proposal succeeds, what tangible benefits will it add?

Q6. What are the risks?

Q7. How long will it take?

Q8. What are the midterm and final “exams” to check for success?

May not do  justice to your proposal.



   view my Linkedin profile


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

On Tue, 31 Jan 2023 at 17:35, kazuyuki tanimura <ktanim...@apple.com.invalid>

> Hi everyone,
> I would like to start a discussion on “Lazy Materialization for Parquet
> Read Performance Improvement"
> Chao and I propose a Parquet reader with lazy materialization. For
> Spark-SQL filter operations, evaluating the filters first and lazily
> materializing only the used values can save computation wastes and improve
> the read performance.
> The current implementation of Spark requires the read values to
> materialize (i.e. decompress, de-code, etc...) onto memory first before
> applying the filters even though the filters may eventually throw away many
> values.
> We made our design doc as follows.
> SPIP Jira: https://issues.apache.org/jira/browse/SPARK-42256
> SPIP Doc:
> https://docs.google.com/document/d/1Kr3y2fVZUbQXGH0y8AvdCAeWC49QJjpczapiaDvFzME
> Liang-Chi was kind enough to shepherd this effort.
> Thank you
> Kazu

Reply via email to