----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19520/#review38183 -----------------------------------------------------------
client/src/main/java/org/apache/oozie/cli/OozieCLI.java <https://reviews.apache.org/r/19520/#comment70168> RestConstants.LOG_STREAMING_PARAMETER > RestConstants.LOG_FILTER_OPTION. Can you make the option -logfilter instead of -param and refer it as log filter everywhere. Option is used with job command and -param is very generic and does not clarify what it applies to. Command line help should be more informative. List out the options of the -logfilter and usage. Refer to FILTER_OPTION in OozieCLI client/src/main/java/org/apache/oozie/cli/OozieCLI.java <https://reviews.apache.org/r/19520/#comment70169> formatting client/src/main/java/org/apache/oozie/client/rest/RestConstants.java <https://reviews.apache.org/r/19520/#comment70170> LOG_FILTER ="logfilter" core/src/main/java/org/apache/oozie/BundleEngine.java <https://reviews.apache.org/r/19520/#comment70173> LogUserParam -> LogFilter core/src/main/java/org/apache/oozie/BundleEngine.java <https://reviews.apache.org/r/19520/#comment70177> Add this as isTerminalStatus() to BundleJobBean. Can copy from CoordinatorJobBean core/src/main/java/org/apache/oozie/BundleEngine.java <https://reviews.apache.org/r/19520/#comment70176> Keep a buffer of two minutes to pick up log statements after db update command to update last modified time was done. core/src/main/java/org/apache/oozie/CoordinatorEngine.java <https://reviews.apache.org/r/19520/#comment70178> buffer of 2 mins core/src/main/java/org/apache/oozie/CoordinatorEngine.java <https://reviews.apache.org/r/19520/#comment70182> Remove the commented line core/src/main/java/org/apache/oozie/DagEngine.java <https://reviews.apache.org/r/19520/#comment70183> startTime variable is not used and can be removed. job.getCreatedTime() can directly be passed as before core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java <https://reviews.apache.org/r/19520/#comment70184> Can you add a toString() method to the filter class and print details with that instead of doing it here? core/src/main/java/org/apache/oozie/util/LogUserParam.java <https://reviews.apache.org/r/19520/#comment70186> Can we merge this class and XLogStreamer.Filter into one LogFilter class? It also has log levels, etc core/src/main/java/org/apache/oozie/util/LogUserParam.java <https://reviews.apache.org/r/19520/#comment70187> Don't think this is good. Action ids will have @. There will be lot of -, [, ], etc in log statements. May be we can just avoid wildcard (*) for now or not put any restriction as we can put limit on the max time range scanned now. core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java <https://reviews.apache.org/r/19520/#comment70208> DateFormat is not thread safe. Either use threadlocal or use local variable core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java <https://reviews.apache.org/r/19520/#comment70209> parseException core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java <https://reviews.apache.org/r/19520/#comment70212> Please handle all this logic of loglimit and enddate in filter.matches() itself. core/src/main/java/org/apache/oozie/util/XLogStreamer.java <https://reviews.apache.org/r/19520/#comment70213> formatting core/src/main/java/org/apache/oozie/util/XLogStreamer.java <https://reviews.apache.org/r/19520/#comment70224> Can we move the logic of calculating start time and end time to LogFilter class itself getLogScanStartTime(String jobStartTime, String jobEndTime) getLogScanEndTime(String jobStartTime, String jobEndTime) All the logic for filtering should only be in Filter class. They should not spill into XLogStreamer core/src/main/java/org/apache/oozie/util/XLogStreamer.java <https://reviews.apache.org/r/19520/#comment70238> We need to do this only for bundle or coord job (with no actions). If they give a list of actions it is better to do a longer time range in one scan for all the actions. core/src/main/resources/oozie-default.xml <https://reviews.apache.org/r/19520/#comment70239> max.log.scan.duration docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70242> Filtering the server logs with logfilter options docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70240> make log streaming docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70276> Explain what offset for start and end do. Also tell what is the default when not specified. Give short examples here itself. i.e something like For eg: start=2013-01-01T05:00Z or start=1h or start=20m Saw most of this is in conditions and usage. Better to merge them here as much as possible so that they are closer to the context. I also see lot of split lines in the bulleted list. Did you check the html page generated. It will screw up bullets if the whole text is not in the same lien docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70277> Provide all options docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70261> Can we call this numlines instead of limit? patten -> pattern. Can you run your patch through a spell checker? docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70278> Prints debug information on the log files, time ranges and patterns being searched for. Can be used to debug if expected logs are not shown with the filter options provided. docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70281> Can remove this. docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70274> Can you combine text and missing example. i.e "Missing" in last 1 hr. docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70275> Believe length and limit are same. 1st example has length. This example can be removed docs/src/site/twiki/DG_CommandLineTool.twiki <https://reviews.apache.org/r/19520/#comment70273> Is the user required to specify millis for start and end? Can we use the format for start and end time range as rest of the other commands. i.e 2011-12-01T05:00Z webapp/src/main/webapp/oozie-console.js <https://reviews.apache.org/r/19520/#comment70268> logFilter webapp/src/main/webapp/oozie-console.js <https://reviews.apache.org/r/19520/#comment70269> Errored out webapp/src/main/webapp/oozie-console.js <https://reviews.apache.org/r/19520/#comment70272> Can we populate initial screen of logs with the list of options for log filter? webapp/src/main/webapp/oozie-console.js <https://reviews.apache.org/r/19520/#comment70270> Get Logs - Rohini Palaniswamy On March 21, 2014, 4:55 p.m., Purshotam Shah wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/19520/ > ----------------------------------------------------------- > > (Updated March 21, 2014, 4:55 p.m.) > > > Review request for oozie. > > > Bugs: OOZIE-1737 > https://issues.apache.org/jira/browse/OOZIE-1737 > > > Repository: oozie-git > > > Description > ------- > > User can provide multiple option as -param <>;<>;<> to make streaming faster > and precise. > Option can be. > * Recent: Define offset of recent hours/Min of log to stream. Server will > search user query only in recent log duration. > * Start: Start time of log parsing. Oozie server will start parsing logs > from start date. User can provide a valid date or > offset. > * End: Server will streaming logs till specified end date. User can > provide a valid date or offset. > * Loglevel : Multiple log levels separated by "|" can be specified. > * Text: String to search in logs. > * Limit : Limit number of line to be searched. Log search will end when > when n lines(excluding exception) of patten has > matched. > * Debug : Will print log start and end time. This will be useful, if user > are not sure what his/her offset will be > calculated to. > > > In addition to this > 1. System setting to control log streaming duration. > 2. Added a option to show log streaming status. > A label on UI will show if log streaming is in progress, completed or > erred. > 3. Many user has complied that get button doesn't look like button. Changes > the look of "get log" button. > 4. End the execution when data is date is reached. > earlier started date and end date is only used to load logs files. > ...... > > > Diffs > ----- > > client/src/main/java/org/apache/oozie/cli/OozieCLI.java 3c780dd > client/src/main/java/org/apache/oozie/client/OozieClient.java b0a85fd > client/src/main/java/org/apache/oozie/client/rest/RestConstants.java > 0466ffe > core/src/main/java/org/apache/oozie/BundleEngine.java ce7c9c4 > core/src/main/java/org/apache/oozie/CoordinatorEngine.java 315a330 > core/src/main/java/org/apache/oozie/DagEngine.java 300d6eb > core/src/main/java/org/apache/oozie/service/XLogStreamingService.java > f77794e > core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java > 8dc8b4b > core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 2ef351e > core/src/main/java/org/apache/oozie/util/LogUserParam.java e69de29 > core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java > 8a11780 > core/src/main/java/org/apache/oozie/util/XLogStreamer.java bc6b009 > core/src/main/resources/oozie-default.xml 34362aa > core/src/test/java/org/apache/oozie/TestCoordinatorEngineStreamLog.java > b4f161a > core/src/test/java/org/apache/oozie/service/TestLogStreamingParam.java > e69de29 > core/src/test/resources/userParamTestlog.log e69de29 > docs/src/site/twiki/DG_CommandLineTool.twiki 0748ff8 > webapp/src/main/webapp/oozie-console.js 0b6cedf > > Diff: https://reviews.apache.org/r/19520/diff/ > > > Testing > ------- > > > Thanks, > > Purshotam Shah > >
