> On Aug. 19, 2013, 5:04 a.m., Rohini Palaniswamy wrote: > > /trunk/core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java, > > line 138 > > <https://reviews.apache.org/r/11922/diff/9/?file=340665#file340665line138> > > > > Logic is good, but implementation needs rethinking as it is not > > efficient. Can be a separate jira. > > > > Few things that I could think of > > > > 1) If you are reading over http and at the same time > > processing/filtering on this server and holding up the connection, based on > > tomcat timeout the connection to other servers will mostly get dropped. > > > > 2) Processing is going to be very slow with the treemap implementation > > and the Oozie UI Ajax call could time out. > > > > > > Robert Kanter wrote: > 1) Is there some way to keep the connection alive longer? Otherwise, the > only alternative I can think of is to bring all of the logs into memory like > I was doing before, but this has other issues (i.e. using too much memory). > > 2) The TreeMap should only have at most N items in it at any given time, > where N is the number of servers. So I don't think this should be too slow; > in any case, we need to keep the N items sorted and this seemed like a good > way -- what would be a better way? > > Rohini Palaniswamy wrote: > Haven't given much thought. One thing that comes to mind is to stream > files in temporary directory first and then merge them instead of keeping it > in memory. > > The connection timeout setting is a tomcat setting and cannot be changed > in application. With or without timeouts (infinite timeout), making multiple > IO calls to read line by line from server itself will introduce lot of > slowness. If you address 1) by downloading in bulk and reading locally 2) > might not be that much of a problem.
Unfortunately, I don't see any alternatives to reading/processing line-by-line because we have to properly order the messages. But I think you are right and we should stream the files to a temp dir first and then read them locally. I'll work on doing this. As for the overall approach, we currently read line-by-line without HA in the XLogReader class. If we really wanted to improve this whole feature, then I think we'd have to completely rework how logs are written (e.g. one log file per job id), regardless of HA. - Robert ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11922/#review23518 ----------------------------------------------------------- On Aug. 13, 2013, 5:35 p.m., Robert Kanter wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11922/ > ----------------------------------------------------------- > > (Updated Aug. 13, 2013, 5:35 p.m.) > > > Review request for oozie. > > > Bugs: OOZIE-615 > https://issues.apache.org/jira/browse/OOZIE-615 > > > Repository: oozie > > > Description > ------- > > See this comment for details: > https://issues.apache.org/jira/browse/OOZIE-615?focusedCommentId=13686181&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13686181 > > > Diffs > ----- > > /trunk/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1513595 > /trunk/client/src/main/java/org/apache/oozie/client/OozieClient.java > 1513595 > /trunk/client/src/main/java/org/apache/oozie/client/rest/RestConstants.java > 1513595 > /trunk/core/pom.xml 1513595 > /trunk/core/src/main/java/org/apache/oozie/BaseEngine.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/BundleEngine.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/CoordinatorEngine.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/DagEngine.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/ErrorCode.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/command/Command.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1513595 > > /trunk/core/src/main/java/org/apache/oozie/service/ActionCheckerService.java > 1513595 > > /trunk/core/src/main/java/org/apache/oozie/service/JobsConcurrencyService.java > PRE-CREATION > /trunk/core/src/main/java/org/apache/oozie/service/MemoryLocksService.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/service/PauseTransitService.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/service/PurgeService.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java > 1513595 > > /trunk/core/src/main/java/org/apache/oozie/service/StatusTransitService.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/service/XLogService.java 1513595 > > /trunk/core/src/main/java/org/apache/oozie/service/XLogStreamingService.java > PRE-CREATION > > /trunk/core/src/main/java/org/apache/oozie/service/ZKJobsConcurrencyService.java > PRE-CREATION > /trunk/core/src/main/java/org/apache/oozie/service/ZKLocksService.java > PRE-CREATION > > /trunk/core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java > PRE-CREATION > /trunk/core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/servlet/ServicesLoader.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java > 1513595 > /trunk/core/src/main/java/org/apache/oozie/util/ConfigUtils.java 1513595 > > /trunk/core/src/main/java/org/apache/oozie/util/FixedJsonInstanceSerializer.java > PRE-CREATION > /trunk/core/src/main/java/org/apache/oozie/util/LockToken.java PRE-CREATION > /trunk/core/src/main/java/org/apache/oozie/util/MemoryLocks.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/util/MultiFileReader.java > PRE-CREATION > > /trunk/core/src/main/java/org/apache/oozie/util/SimpleTimestampedMessageParser.java > PRE-CREATION > > /trunk/core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java > PRE-CREATION > /trunk/core/src/main/java/org/apache/oozie/util/XLogReader.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/util/XLogStreamer.java 1513595 > /trunk/core/src/main/java/org/apache/oozie/util/ZKUtils.java PRE-CREATION > /trunk/core/src/main/resources/oozie-default.xml 1513595 > > /trunk/core/src/test/java/org/apache/oozie/TestCoordinatorEngineStreamLog.java > 1513595 > /trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java > 1513595 > > /trunk/core/src/test/java/org/apache/oozie/service/DummyLogStreamingServlet.java > PRE-CREATION > > /trunk/core/src/test/java/org/apache/oozie/service/TestJobsConcurrencyService.java > PRE-CREATION > /trunk/core/src/test/java/org/apache/oozie/service/TestXLogService.java > 1513595 > > /trunk/core/src/test/java/org/apache/oozie/service/TestXLogStreamingService.java > PRE-CREATION > > /trunk/core/src/test/java/org/apache/oozie/service/TestZKJobsConcurrencyService.java > PRE-CREATION > /trunk/core/src/test/java/org/apache/oozie/service/TestZKLocksService.java > PRE-CREATION > > /trunk/core/src/test/java/org/apache/oozie/service/TestZKXLogStreamingService.java > PRE-CREATION > > /trunk/core/src/test/java/org/apache/oozie/servlet/MockCoordinatorEngineService.java > 1513595 > > /trunk/core/src/test/java/org/apache/oozie/servlet/MockDagEngineService.java > 1513595 > /trunk/core/src/test/java/org/apache/oozie/test/ZKXTestCase.java > PRE-CREATION > /trunk/core/src/test/java/org/apache/oozie/util/TestLogStreamer.java > 1513595 > /trunk/core/src/test/java/org/apache/oozie/util/TestMemoryLocks.java > 1513595 > /trunk/core/src/test/java/org/apache/oozie/util/TestMultiFileReader.java > PRE-CREATION > > /trunk/core/src/test/java/org/apache/oozie/util/TestSimplifiedTimestampedMessageParser.java > PRE-CREATION > > /trunk/core/src/test/java/org/apache/oozie/util/TestTimestampedMessageParser.java > PRE-CREATION > /trunk/core/src/test/java/org/apache/oozie/util/TestXLogFilter.java 1513595 > /trunk/core/src/test/java/org/apache/oozie/util/TestXLogReader.java 1513595 > /trunk/core/src/test/java/org/apache/oozie/util/TestZKUtils.java > PRE-CREATION > /trunk/distro/src/main/tomcat/ssl-web.xml 1513595 > /trunk/docs/src/site/twiki/AG_Install.twiki 1513595 > /trunk/docs/src/site/twiki/DG_CommandLineTool.twiki 1513595 > /trunk/docs/src/site/twiki/WebServicesAPI.twiki 1513595 > /trunk/pom.xml 1513595 > > Diff: https://reviews.apache.org/r/11922/diff/ > > > Testing > ------- > > > Thanks, > > Robert Kanter > >
