[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17453199#comment-17453199 ] Raymond Xu commented on HUDI-2151: -- Codec change: change default codec to SNAPPY instead of GZIP * it's

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440051#comment-17440051 ] Vinoth Chandar commented on HUDI-2151: -- [~shivnarayan]  this is actually super high priority. lets

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436219#comment-17436219 ] Prashant Wason commented on HUDI-2151: -- >> Is file listing parallelism too high? In my testing, 256

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398982#comment-17398982 ] ASF GitHub Bot commented on HUDI-2151: -- nsivabalan commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398981#comment-17398981 ] ASF GitHub Bot commented on HUDI-2151: -- nsivabalan merged pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398944#comment-17398944 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398930#comment-17398930 ] ASF GitHub Bot commented on HUDI-2151: -- vinothchandar commented on a change in pull request #3452:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398926#comment-17398926 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398920#comment-17398920 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398911#comment-17398911 ] ASF GitHub Bot commented on HUDI-2151: -- nsivabalan commented on a change in pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398631#comment-17398631 ] ASF GitHub Bot commented on HUDI-2151: -- nsivabalan commented on a change in pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398344#comment-17398344 ] ASF GitHub Bot commented on HUDI-2151: -- vinothchandar commented on a change in pull request #3452:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398308#comment-17398308 ] ASF GitHub Bot commented on HUDI-2151: -- umehrot2 commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398245#comment-17398245 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398215#comment-17398215 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398214#comment-17398214 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3452: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398171#comment-17398171 ] ASF GitHub Bot commented on HUDI-2151: -- vinothchandar commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397968#comment-17397968 ] ASF GitHub Bot commented on HUDI-2151: -- danny0405 commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395780#comment-17395780 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395616#comment-17395616 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393922#comment-17393922 ] ASF GitHub Bot commented on HUDI-2151: -- umehrot2 commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393528#comment-17393528 ] ASF GitHub Bot commented on HUDI-2151: -- umehrot2 commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391863#comment-17391863 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391903#comment-17391903 ] ASF GitHub Bot commented on HUDI-2151: -- yanghua commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391852#comment-17391852 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot edited a comment on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391850#comment-17391850 ] ASF GitHub Bot commented on HUDI-2151: -- hudi-bot commented on pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391849#comment-17391849 ] ASF GitHub Bot commented on HUDI-2151: -- umehrot2 opened a new pull request #3383: URL:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380369#comment-17380369 ] Vinoth Chandar commented on HUDI-2151: -- Async compaction should be off by default? for batch writer?

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380368#comment-17380368 ] Vinoth Chandar commented on HUDI-2151: -- Hive timestamp support?   {code:java}   val

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380367#comment-17380367 ] Vinoth Chandar commented on HUDI-2151: -- How do we reduce pain around Hive sync?   {code:java} val

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380336#comment-17380336 ] Vinoth Chandar commented on HUDI-2151: -- Row writing   {code:java} val ENABLE_ROW_WRITER_OPT_KEY:

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380332#comment-17380332 ] Vinoth Chandar commented on HUDI-2151: -- Should have hive style partitioning by default. but is it

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380210#comment-17380210 ] Vinoth Chandar commented on HUDI-2151: -- Should this be smaller than the target file size? Would n't

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380198#comment-17380198 ] Vinoth Chandar commented on HUDI-2151: -- marker based rollback should be on   {code:java} public

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380196#comment-17380196 ] Vinoth Chandar commented on HUDI-2151: -- Should these be true?   {code:java} public static final

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380195#comment-17380195 ] Vinoth Chandar commented on HUDI-2151: -- These parallelisms seem low   {code:java} public static

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380194#comment-17380194 ] Vinoth Chandar commented on HUDI-2151: -- Dynamic bloom filters.   {code:java} public static final

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380190#comment-17380190 ] Vinoth Chandar commented on HUDI-2151: -- Should this be higher? If typical record sizes are large? (or

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380181#comment-17380181 ] Vinoth Chandar commented on HUDI-2151: -- Should n't we have a default index type?   {code:java}

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380174#comment-17380174 ] Vinoth Chandar commented on HUDI-2151: -- Need to ensure this is actually 1, going forward.  

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380173#comment-17380173 ] Vinoth Chandar commented on HUDI-2151: -- Should this be spillable map?   {code:java} public static

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379623#comment-17379623 ] Vinoth Chandar commented on HUDI-2151: -- Ensure the 1024 is not blindly used always? {code:java}  

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379619#comment-17379619 ] Vinoth Chandar commented on HUDI-2151: -- Make this lazy? {code:java}   public static final

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379617#comment-17379617 ] Vinoth Chandar commented on HUDI-2151: -- Why would we even want to do lazyRead=false anymore?    

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379615#comment-17379615 ] Vinoth Chandar commented on HUDI-2151: -- Switch to default payload?   {code:java} public static final

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379496#comment-17379496 ] Vinoth Chandar commented on HUDI-2151: -- These need a default value.    {code:java} public static

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379488#comment-17379488 ] Vinoth Chandar commented on HUDI-2151: -- Is this correct? 5s?   {code:java} public static final

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379161#comment-17379161 ] Vinoth Chandar commented on HUDI-2151: -- multi writer should be turned on    {code:java} public

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378873#comment-17378873 ] Vinoth Chandar commented on HUDI-2151: -- Should this be turned to true?   {code:java}   public static

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378870#comment-17378870 ] Vinoth Chandar commented on HUDI-2151: -- Should probably process things async? Need to look at code

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378152#comment-17378152 ] Vinoth Chandar commented on HUDI-2151: -- Does this even help?   {code:java}  public static final

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377669#comment-17377669 ] Vinoth Chandar commented on HUDI-2151: -- Should we turn on avro schema validation   {code:java}

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377663#comment-17377663 ] Vinoth Chandar commented on HUDI-2151: -- Rollback using markers {code:java} public static final

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377665#comment-17377665 ] Vinoth Chandar commented on HUDI-2151: -- Timeline layout version should now be 1   {code:java} public

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377618#comment-17377618 ] Vinoth Chandar commented on HUDI-2151: -- Metadata table needs to be turned on.   {code:java} public

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377617#comment-17377617 ] Vinoth Chandar commented on HUDI-2151: -- Is file listing parallelism too high? {code:java} public

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377615#comment-17377615 ] Vinoth Chandar commented on HUDI-2151: -- Do we really need optimisitic consistency guard turned on?