GitHub user jermjensen added a comment to the discussion: Direct Pipeline with No Transforms
Thanks for the follow up - it doesn't seem like anything I can do will affect the fetch size of the source system. I've been doing some digging, and I think I know what the issue is - maybe you guys can help shed some light. The goal is to limit/meter the number of rows coming from the source system. I've been focusing on trying to set the fetch size of the driver. The source system I'm connecting to is an old Sybase server - and the default "fetch size" parameter for Sybase is 0. Apparently, this default behavior is to load all of the data returned by the SELECT into memory then move it down to Hop. It looks like this default is different from modern JDBC defaults. There is an option for setting a fetch size in the JDBC connection, but it can be overridden if at any point in the Java code a setFetchSize method is called. I looked at the code searching for a call to setFetchSize in Hop, but I didn't find anything that might be triggering this behavior. Could the team speak to how fetch sizes are defaulted in Hop and if the behavior of this driver - that calling setFetchSize(0) returns all data - might be causing my issue? GitHub link: https://github.com/apache/hop/discussions/4872#discussioncomment-12097778 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
