Re: Review Request 64976: Ambari Infra Manager: hdfs upload support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64976/#review194927 --- Ship it! Ship It! ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java Lines 35 (patched) <https://reviews.apache.org/r/64976/#comment274013> mostly in ambari code, we are not using * in imports (convention) - Oliver Szabo On Jan. 5, 2018, 2:43 p.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64976/ > --- > > (Updated Jan. 5, 2018, 2:43 p.m.) > > > Review request for Ambari, Attila Doroszlai, Oliver Szabo, and Sid Wagle. > > > Bugs: AMBARI-22731 > https://issues.apache.org/jira/browse/AMBARI-22731 > > > Repository: ambari > > > Description > --- > > * while arhiving documents from solr result files can be saved to hdfs > * s3 and hdfs releated parameters can be overwritten using infra manger api > * destination directory name contains the interval end value when archiving > to local filesystem > * job can be restarted using infra manger api > * bugfix: infra manger startup failed when no jobs were defined in the > properties > * IT: test for archiving to hdfs > * IT: test for archiving to local filesystem > * vagrant-infra-manager.properties file for integration testing with an > ambari installed cluster > > > Diffs > - > > ambari-infra/ambari-infra-manager-it/pom.xml 97e8ea0f44 > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java > 0e391a3e6e > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java > f48d4c2b49 > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java > 22826a01fc > > ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story > 1044258816 > ambari-infra/ambari-infra-manager/docker/docker-compose.yml 1172631917 > ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh > 6ddb7c292a > ambari-infra/ambari-infra-manager/pom.xml de131b03f6 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/JobProperties.java > 0841dd7fd9 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentArchivingConfiguration.java > 7588b9992a > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java > 1484eedfe3 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/ExportDestination.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileNameSuffixFormatter.java > af48ab9aaa > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HdfsUploader.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/deleting/DocumentDeletingConfiguration.java > 4fce4b677f > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/model/JobExecutionRestartRequest.java > 88687e7260 > > ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties > 9103d09e7e > > ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/JobPropertiesTest.java > 6a7622954f > > ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/FileNameSuffixFormatterTest.java > 34e679f298 > > ambari-infra/ambari-infra-manager/src/test/resoruces/vagrant-infra-manager.properties > PRE-CREATION > ambari-infra/pom.xml 9e7a71bfa7 > > > Diff: https://reviews.apache.org/r/64976/diff/1/ > > > Testing > --- > > * unit tests > * integration test using docker > * integration test using vagrant, ambari and ide > > > Thanks, > > Krisztian Kasa > >
Re: Review Request 64852: Ambari Infra Manager: deleting documents
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64852/#review194580 --- Ship it! Ship It! - Oliver Szabo On Dec. 28, 2017, 2:09 p.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64852/ > --- > > (Updated Dec. 28, 2017, 2:09 p.m.) > > > Review request for Ambari, Attila Magyar, Oliver Szabo, and Sid Wagle. > > > Bugs: AMBARI-22653 and AMBARI-22702 > https://issues.apache.org/jira/browse/AMBARI-22653 > https://issues.apache.org/jira/browse/AMBARI-22702 > > > Repository: ambari > > > Description > --- > > * defining new job for deleting documents from solr: collection and filter > field can be defined in properties > * archiving job supports deleting the succesfully uploaded documents > * date time format in output filename can be defined in property file > * s3 secret and access keys can be defined in a csv file and this file can be > referenced from infra-manager.properties or in environmental variables > * generic property processing for jobs > * Helper class for processing parametrized solr query text > * new IT case: Running archiving job with a bigger start value than end value > exports and deletes 0 documents > * new IT case: After Deleting job deletes documents from solr no document > found in the specified interval > * existing IT cases for archiving also check that solr has no more documents > left in the specified interval after archiving job finished > > > Diffs > - > > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java > 9db562cec3 > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java > 703e1cfa67 > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java > 4a09d7d7ee > > ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story > cd1f49d052 > ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh > ab026598b1 > ambari-infra/ambari-infra-manager/pom.xml 67bf7d133b > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/JobProperties.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/PropertyMap.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/SolrDAOBase.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java > 7a30393224 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java > 8421802cd7 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java > 1f3957a034 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java > 189591102b > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java > 3b6c402458 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java > 37f6d1b669 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java > 9af4afc630 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentWiper.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java > d3f2a65e07 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileNameSuffixFormatter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/ItemWriterListener.java > 742777109e > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java > baad61bdcb > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalItemWriterListener.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java > 0979f10bfc > > ambari-infra/ambari-infra-manager/sr
Re: Review Request 64850: Fixing interface and file renaming issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64850/#review194527 --- Ship it! Ship It! - Oliver Szabo On Dec. 27, 2017, 8:10 a.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64850/ > --- > > (Updated Dec. 27, 2017, 8:10 a.m.) > > > Review request for Ambari, Attila Doroszlai, Oliver Szabo, and Sid Wagle. > > > Bugs: AMBARI-22653 > https://issues.apache.org/jira/browse/AMBARI-22653 > > > Repository: ambari > > > Description > --- > > Patch https://reviews.apache.org/r/64607 contains two renamed interfaces: > DocumentIterator -> CloseableIterator > DocumentSource -> ItemWriterListener > > but the files containing these interface definitions were not renamed on > trunk git repo > > > Diffs > - > > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java > > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java > > > > Diff: https://reviews.apache.org/r/64850/diff/1/ > > > Testing > --- > > maven build > unit and integration tests > > > Thanks, > > Krisztian Kasa > >
Review Request 64660: Rafactor: Package Log Search and Log Feeder into jars + cleanup start scripts
pts/logsearch.sh PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/scripts/run.sh 241b32f ambari-logsearch/ambari-logsearch-server/src/main/scripts/stop.sh 3612f08 ambari-logsearch/docker/Dockerfile e79f61b ambari-logsearch/docker/all.yml 369db39 ambari-logsearch/docker/bin/start.sh bea06de ambari-logsearch/docker/docker-compose.yml 725af36 ambari-logsearch/docker/logsearch-logfeeder.yml 4d48501 ambari-logsearch/docker/logsearch-server.yml 6a0d71e ambari-logsearch/docker/test-config/logfeeder/logfeeder-env.sh 5beb093 ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 1ae57bb ambari-logsearch/docker/test-config/logsearch/logsearch-env.sh 80f9c4c ambari-logsearch/docker/test-config/logsearch/logsearch-https.properties 5e7853d ambari-logsearch/docker/test-config/logsearch/logsearch.properties 5bde17c ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml e308479 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-properties.xml a38f961 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml 5c263db ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logfeeder.py fbc29c8 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch.py 4410712 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_common.py c4e0c16 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py 884d32a ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py 653d604 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py 7738cc1 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logfeeder-env.sh.j2 f2dd06b ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-env.sh.j2 7d9b5de ambari-server/src/main/resources/stack-hooks/after-INSTALL/scripts/params.py bf9d79b ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 19c81a8 ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 159cb81 ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logsearch.py db19e0f ambari-server/src/test/python/stacks/2.4/configs/default.json 9966f3e Diff: https://reviews.apache.org/r/64660/diff/1/ Testing --- UTs passed after some changes. FT: - install packages manually with yum (try out that i have keys + checkpoint folder in old /etc/.. conf locations) - start applications with docker compose + start from IDE as well. - tried out logfeeder/logsearch script options (start/stop/status/help/test) Thanks, Oliver Szabo
Re: Review Request 64607: Ambari Infra Manager: solr data exporting jobs and integration test environment
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64607/#review193910 --- Ship it! Shipt It! - Oliver Szabo On Dec. 15, 2017, 8:57 a.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64607/ > --- > > (Updated Dec. 15, 2017, 8:57 a.m.) > > > Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle. > > > Bugs: AMBARI-22514 and AMBARI-22653 > https://issues.apache.org/jira/browse/AMBARI-22514 > https://issues.apache.org/jira/browse/AMBARI-22653 > > > Repository: ambari > > > Description > --- > > * Integration test environment for ambari-infra-manager using docker-compose > and jbehave > * test cases for testing data export jobs > > * deleting temporary files as soon as the file becomes unnecessary > * configuring solr data export like jobs in property files only, no hardcoded > job and step configurations: query can be defined in property file for > pulling data from solr > * validate job property values at application startup and before job execution > * s3 upload support > * not just interval end but interval start can be specified when exporting > * interval start and end values can be other type than dateTime > > > Diffs > - > > ambari-infra/ambari-infra-manager-it/pom.xml PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java > PRE-CREATION > ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story > PRE-CREATION > ambari-infra/ambari-infra-manager/docker/Dockerfile adb584acec > ambari-infra/ambari-infra-manager/docker/docker-compose.yml PRE-CREATION > ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java > 84ce16074b > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java > 84f5ece26f > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java > 69f41d333e > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java > f1df46cfcc > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java > d6301c0c0e > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java > 3bab6d505e > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java > a4378a4340 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java > 6232cfc4c6 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java > c9871a3660 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java > 26a8c6310f > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java > 02d898d979 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java > 495401d633 > > a
Re: Review Request 64607: Ambari Infra Manager: solr data exporting jobs and integration test environment
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64607/#review193796 --- ambari-infra/ambari-infra-manager-it/pom.xml Lines 111 (patched) <https://reviews.apache.org/r/64607/#comment272445> is it skipped by default if you use "mvn integration-test" so skipTests applied here properly (i think it only affects UTs), probably you should pust this plugin usage into a profile (so we can enable it with -Pit or something) - Oliver Szabo On Dec. 14, 2017, 12:58 p.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64607/ > --- > > (Updated Dec. 14, 2017, 12:58 p.m.) > > > Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle. > > > Bugs: AMBARI-22514 and AMBARI-22653 > https://issues.apache.org/jira/browse/AMBARI-22514 > https://issues.apache.org/jira/browse/AMBARI-22653 > > > Repository: ambari > > > Description > --- > > * Integration test environment for ambari-infra-manager using docker-compose > and jbehave > * test cases for testing data export jobs > > * deleting temporary files as soon as the file becomes unnecessary > * configuring solr data export like jobs in property files only, no hardcoded > job and step configurations: query can be defined in property file for > pulling data from solr > * validate job property values at application startup and before job execution > * s3 upload support > * not just interval end but interval start can be specified when exporting > * interval start and end values can be other type than dateTime > > > Diffs > - > > ambari-infra/ambari-infra-manager-it/pom.xml PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java > PRE-CREATION > ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties > PRE-CREATION > > ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story > PRE-CREATION > ambari-infra/ambari-infra-manager/docker/Dockerfile adb584acec > ambari-infra/ambari-infra-manager/docker/docker-compose.yml PRE-CREATION > ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java > 84ce16074b > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java > 84f5ece26f > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java > 69f41d333e > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java > f1df46cfcc > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java > d6301c0c0e > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java > 3bab6d505e > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java > a4378a4340 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java > 6232cfc4c6 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java > c9871a3660 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job
Re: Review Request 64553: Log Feeder refactor: integrate with spring boot
> On Dec. 13, 2017, 9:18 p.m., Sid Wagle wrote: > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java > > Lines 40 (patched) > > <https://reviews.apache.org/r/64553/diff/4/?file=1915301#file1915301line40> > > > > When would this be false? in the ApplicationConfig, then the object will be ready during setting up application context, and an other one can start the iniialization > On Dec. 13, 2017, 9:18 p.m., Sid Wagle wrote: > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java > > Lines 30 (patched) > > <https://reviews.apache.org/r/64553/diff/4/?file=1915329#file1915329line30> > > > > More contemporary design would be to create a Runnable that can be used > > by the caller as an argument scheduledExectuor. The scehduling logic can be > > kept out of the class. as i mentioned Krisztina that patch focus on spring boot support so that class was just moved here, probably we will get rid of it - Oliver --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64553/#review193736 ------- On Dec. 13, 2017, 3:24 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64553/ > --- > > (Updated Dec. 13, 2017, 3:24 p.m.) > > > Review request for Ambari, Krisztian Kasa and Sid Wagle. > > > Bugs: AMBARI-22639 > https://issues.apache.org/jira/browse/AMBARI-22639 > > > Repository: ambari > > > Description > --- > > First patch for refactor logfeeder - Spring Boot support > > Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil > and let Spring to handle configs & beans. > - remove LogFeederPropertiesUtil > - move logfeeder properties fields into mutliple POJOs > - update some tests & remove one test (needs to rewrite it) > - convert singleton classes to spring beans > - let Spring to handle application pid file > - convert some internal threads to classes (like StatsLogger) > > (it does not contain any real changes regarding to input-filter-output blocks) > > Next patch will be focus on the usage of start scritps (in ambari-logsearch > and ambari-server stack code) > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java > a84a97b > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java > 6ed36fd > ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION > ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java > 5114743 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java > d996f98 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java > 243b344 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java > 5c20a8e > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java > ec29f69 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java > a76 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/ApplicationConfig.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/InputSimulateConfig.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogEntryCacheConfig.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/MetricsCollectorConfig.java > PRE-CREATION > > ambari-logsearch/ambari-logs
Re: Review Request 64553: Log Feeder refactor: integrate with spring boot
> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
> > Line 62 (original), 64 (patched)
> > <https://reviews.apache.org/r/64553/diff/3/?file=1914915#file1914915line64>
> >
> > Create an abstract init method and call it from here. The comment "This
> > method needs to be overwritten by deriving classes." would be unnecessary.
> > Like:
> >
> > protected abstract onInitializing(LogFeederProps logFeederProps);
> >
> > public void init(LogFeederProps logFeederProps) throws Exception {
> > this.logFeederProps = logFeederProps;
> > onInitializing(logFeederProps);
> > }
you are right but as i wrote it does not contain any real changes on the blocks
part (focus is on make it work with spring boot), so these will be refactored
in different patch
> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
> > Lines 213 (patched)
> > <https://reviews.apache.org/r/64553/diff/3/?file=1914921#file1914921line213>
> >
> > Are the same properties are stored here as the fields of this class?
no its all properties, so you can pass all of them if you want through this
class
> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
> > Lines 160 (patched)
> > <https://reviews.apache.org/r/64553/diff/3/?file=1914922#file1914922line160>
> >
> > Isn't it isEmpty?
yes, thanks
> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java
> > Lines 64 (patched)
> > <https://reviews.apache.org/r/64553/diff/3/?file=1914936#file1914936line64>
> >
> > Wouldn't be better to halt the entire application in case of an Error?
> > So Errors would be rethrown but Exceptions not.
regarding the 2 statLogger comment: probably it will be refactored much more
(or i will get rid of this), but now i just moved the class (but review board
show as a newly added one)
- Oliver
-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64553/#review193647
---
On Dec. 13, 2017, 3:24 p.m., Oliver Szabo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64553/
> ---
>
> (Updated Dec. 13, 2017, 3:24 p.m.)
>
>
> Review request for Ambari, Krisztian Kasa and Sid Wagle.
>
>
> Bugs: AMBARI-22639
> https://issues.apache.org/jira/browse/AMBARI-22639
>
>
> Repository: ambari
>
>
> Description
> ---
>
> First patch for refactor logfeeder - Spring Boot support
>
> Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil
> and let Spring to handle configs & beans.
> - remove LogFeederPropertiesUtil
> - move logfeeder properties fields into mutliple POJOs
> - update some tests & remove one test (needs to rewrite it)
> - convert singleton classes to spring beans
> - let Spring to handle application pid file
> - convert some internal threads to classes (like StatsLogger)
>
> (it does not contain any real changes regarding to input-filter-output blocks)
>
> Next patch will be focus on the usage of start scritps (in ambari-logsearch
> and ambari-server stack code)
>
>
> Diffs
> -
>
>
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
> a84a97b
>
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
> 6ed36fd
> ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION
> ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf
>
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
> 5114743
>
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java
> d996f98
>
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
> 243b344
>
>
Re: Review Request 64553: Log Feeder refactor: integrate with spring boot
r/output/OutputHDFSFile.java ba4d60a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputKafka.java 52fc6f8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java 48716fa ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputS3File.java 5b213e8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java 38219df ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederPropertiesUtil.java 1636653 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SSLUtil.java 6bcaac9 ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log-samples/shipper-conf/input.config-sample.json 4ab2eb2 ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log4j.xml eb20665 ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties 115778b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/filter/FilterGrokTest.java 8d7e86c ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/filter/FilterJSONTest.java acc3d4d ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/filter/FilterKeyValueTest.java ae978fb ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputFileTest.java efebc08 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputManagerTest.java 46fbc3b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandlerTest.java 46abc63 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetricsManagerTest.java f74a80e ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputKafkaTest.java 38d4b8b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputManagerTest.java 5abb720 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputS3FileTest.java 7c6aca2 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java 5ab271a Diff: https://reviews.apache.org/r/64553/diff/4/ Changes: https://reviews.apache.org/r/64553/diff/3-4/ Testing --- unit tests done. Check behavior from IDE + docker-compose env Thanks, Oliver Szabo
Review Request 64553: Log Feeder refactor: integrate with spring boot
ogsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/filter/FilterKeyValueTest.java ae978fb ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputFileTest.java efebc08 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputManagerTest.java 46fbc3b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandlerTest.java 46abc63 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetricsManagerTest.java f74a80e ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputKafkaTest.java 38d4b8b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputManagerTest.java 5abb720 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputS3FileTest.java 7c6aca2 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java 5ab271a Diff: https://reviews.apache.org/r/64553/diff/1/ Testing --- unit tests done. Check behavior from IDE + docker-compose env Thanks, Oliver Szabo
Re: Review Request 64513: Update the default storm log search configuration (use wildcards)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64513/ --- (Updated Dec. 12, 2017, 4:24 p.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Sid Wagle. Changes --- upload the correct patch Bugs: AMBARI-22630 https://issues.apache.org/jira/browse/AMBARI-22630 Repository: ambari Description --- set the storm config defaults for logseach as other teams needs it. this patch is for branch-2.6, but if cannot be delivered in 2.6.1, that can be used in a hotfix Diffs (updated) - ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-logsearch-conf.xml d485017 Diff: https://reviews.apache.org/r/64513/diff/3/ Changes: https://reviews.apache.org/r/64513/diff/2-3/ Testing --- not yet. Thanks, Oliver Szabo
Re: Review Request 64513: Update the default storm log search configuration (use wildcards)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64513/ --- (Updated Dec. 12, 2017, 4:19 p.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Sid Wagle. Changes --- update with the commented block Bugs: AMBARI-22630 https://issues.apache.org/jira/browse/AMBARI-22630 Repository: ambari Description --- set the storm config defaults for logseach as other teams needs it. this patch is for branch-2.6, but if cannot be delivered in 2.6.1, that can be used in a hotfix Diffs (updated) - ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-logsearch-conf.xml d485017 Diff: https://reviews.apache.org/r/64513/diff/2/ Changes: https://reviews.apache.org/r/64513/diff/1-2/ Testing --- not yet. Thanks, Oliver Szabo
Re: Review Request 64531: AMBARI-22633 MapDate provides the date incorrectly when Filter is cloned and used in multi-threads
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64531/#review193537 --- Ship it! Pobably that would be better if we would clone all of the mapper types as well. But that wont hurt so +1 - Oliver Szabo On Dec. 12, 2017, 9:39 a.m., Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64531/ > --- > > (Updated Dec. 12, 2017, 9:39 a.m.) > > > Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle. > > > Bugs: AMBARI-22633 > https://issues.apache.org/jira/browse/AMBARI-22633 > > > Repository: ambari > > > Description > --- > > In AMBARI-22600, we cloned the Filter instance to assign the instance per > child thread, which also clones the map of Mapper. Other Mapper implements > only have String(s) type of fields hence thread-safe, but MapperDate has > SimpleDateFormat type of fields which is known as non thread-safe, so cloning > the Filter which has one or more MapperDate and using them concurrently makes > multi-threads issue. > > Replacing SimpleDateFormat with FastDateFormat in commons-lang3 would resolve > the issue. > > > Diffs > - > > ambari-logsearch/ambari-logsearch-logfeeder/pom.xml a06ae902fc > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperDate.java > 6a7fad7b70 > > > Diff: https://reviews.apache.org/r/64531/diff/1/ > > > Testing > --- > > - existing unit tests > - manual tests (manually deploying to cluster) > > > Thanks, > > Jungtaek Lim > >
Review Request 64513: Update the default storm log search configuration (use wildcards)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64513/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Sid Wagle. Bugs: AMBARI-22630 https://issues.apache.org/jira/browse/AMBARI-22630 Repository: ambari Description --- set the storm config defaults for logseach as other teams needs it. this patch is for branch-2.6, but if cannot be delivered in 2.6.1, that can be used in a hotfix Diffs - ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-logsearch-conf.xml d485017 Diff: https://reviews.apache.org/r/64513/diff/1/ Testing --- not yet. Thanks, Oliver Szabo
Review Request 64412: Log Search IT: Expose the right port for Selenium in docker-compose file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64412/ --- Review request for Ambari, Attila Doroszlai, Krisztian Kasa, Sid Wagle, and Istvan Tobias. Bugs: AMBARI-22611 https://issues.apache.org/jira/browse/AMBARI-22611 Repository: ambari Description --- Selenium server uses port not 4004 in docker container. Diffs - ambari-logsearch/docker/all.yml 82f0ff9 Diff: https://reviews.apache.org/r/64412/diff/1/ Testing --- done. Thanks, Oliver Szabo
Re: Review Request 64349: LogFeeder: filter objects for wildcard input paths need to be cloned
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64349/ --- (Updated Dec. 5, 2017, 8:50 p.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Changes --- update summary Summary (updated) - LogFeeder: filter objects for wildcard input paths need to be cloned Bugs: AMBARI-22600 https://issues.apache.org/jira/browse/AMBARI-22600 Repository: ambari Description --- Log Feeder needs to clone filters for different input threads (for wildcard pattern). As different threads tried to access the same grok filter object (mutable) that could cause to some docs used the wrong path (or other fields), especially when a grok error happened. (on error, in the code, we checked some objects regarding that what to do, but other threads changed those objects) Also some other changes: - use sdi_* dynamic fields - add new input log files - add new config to grok filter: skip_on_error (can be useful for filter chains) Diffs - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/Filter.java afd903e ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 49d7e76 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 7df0b6e ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json 34bdcf0 ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log 6a10ad9 ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-2-TestAgg2-4-5/6700/worker.log e69de29 Diff: https://reviews.apache.org/r/64349/diff/1/ Testing --- done, i need to write some unit tests for the filter cloning. Thanks, Oliver Szabo
Review Request 64349: LogFeeder: filters for wildcard input paths need to be cloned
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64349/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22600 https://issues.apache.org/jira/browse/AMBARI-22600 Repository: ambari Description --- Log Feeder needs to clone filters for different input threads (for wildcard pattern). As different threads tried to access the same grok filter object (mutable) that could cause to some docs used the wrong path (or other fields), especially when a grok error happened. (on error, in the code, we checked some objects regarding that what to do, but other threads changed those objects) Also some other changes: - use sdi_* dynamic fields - add new input log files - add new config to grok filter: skip_on_error (can be useful for filter chains) Diffs - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/Filter.java afd903e ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 49d7e76 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 7df0b6e ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json 34bdcf0 ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log 6a10ad9 ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-2-TestAgg2-4-5/6700/worker.log e69de29 Diff: https://reviews.apache.org/r/64349/diff/1/ Testing --- done, i need to write some unit tests for the filter cloning. Thanks, Oliver Szabo
Review Request 64206: Upgrade Ambari Infra to use Solr 5.5.5 for Ambari 2.6.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64206/ --- Review request for Ambari, Attila Doroszlai, Attila Magyar, Krisztian Kasa, and Miklos Gergely. Bugs: AMBARI-22559 https://issues.apache.org/jira/browse/AMBARI-22559 Repository: ambari Description --- upgrade solr version (also use horton repo as default) Diffs - ambari-logsearch/ambari-logsearch-assembly/pom.xml c0a92f4 ambari-logsearch/pom.xml 5a1edee Diff: https://reviews.apache.org/r/64206/diff/1/ Testing --- not required Thanks, Oliver Szabo
Review Request 64156: Log Search: backend should not return with 302 redirect if authentication fails
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64156/ --- Review request for Ambari, Attila Magyar, Krisztian Kasa, and Robert Nettleton. Bugs: AMBARI-22543 https://issues.apache.org/jira/browse/AMBARI-22543 Repository: ambari Description --- Without proper authentication details rest calls returs with 302 (to /login), that should be changed and get a 401 status Diffs - ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/filters/LogsearchAuthenticationEntryPoint.java 2fe5f7b Diff: https://reviews.apache.org/r/64156/diff/1/ Testing --- done Thanks, Oliver Szabo
Re: Review Request 64091: Log Feeder: handle multiple '=' (split char) in key/value filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64091/ --- (Updated Nov. 27, 2017, 8:54 p.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Changes --- split the right values Bugs: AMBARI-22526 https://issues.apache.org/jira/browse/AMBARI-22526 Repository: ambari Description --- Logfeeder key/value filter cannot handle multiple occurence of split char -> in order to solve that we can limit the number of splits Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterKeyValue.java b04a439 Diff: https://reviews.apache.org/r/64091/diff/2/ Changes: https://reviews.apache.org/r/64091/diff/1-2/ Testing --- done Thanks, Oliver Szabo
Review Request 64091: Log Feeder: handle multiple '=' (split char) in key/value filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64091/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22526 https://issues.apache.org/jira/browse/AMBARI-22526 Repository: ambari Description --- Logfeeder key/value filter cannot handle multiple occurence of split char -> in order to solve that we can limit the number of splits Diffs - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterKeyValue.java b04a439 Diff: https://reviews.apache.org/r/64091/diff/1/ Testing --- done Thanks, Oliver Szabo
Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64059/#review191897 --- Ship it! Ship It! - Oliver Szabo On Nov. 27, 2017, 3:39 p.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64059/ > --- > > (Updated Nov. 27, 2017, 3:39 p.m.) > > > Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton. > > > Bugs: AMBARI-22514 > https://issues.apache.org/jira/browse/AMBARI-22514 > > > Repository: ambari > > > Description > --- > > Initial implementation of Schedulable document deletion & archiving for Infra > Solr > == > > Spring bacth job for > > * exporting solr documents from hadoop_logs collection > * saving the documents to filesystem. All document is converted to json and > written into new line. > * compress each file using tar.gz > * job can be started using ambari-infra-manager api and parameters can be > overwritten > * s3 upload support (not used yet) > * job can be configurad to export from any collection: solr query can be > defined in properties > > > Diffs > - > > ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java > fc0a4f71db > > ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties > 81623764da > ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba > > ambari-infra/ambari-infra-manager/src/test/j
Re: Review Request 64083: Log Search: Adding new dynamic field for service logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64083/ --- (Updated Nov. 27, 2017, 3:13 p.m.) Review request for Ambari, Attila Doroszlai, Krisztian Kasa, and Robert Nettleton. Changes --- fix bad refactor Bugs: Ambari https://issues.apache.org/jira/browse/Ambari Repository: ambari Description --- its required to add in order to have a dynamic field which not only indexed, but stored as well. Diffs (updated) - ambari-logsearch/ambari-logsearch-server/src/main/configsets/hadoop_logs/conf/managed-schema c6f498b ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/SolrConstants.java 60fc1a3 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/model/SolrServiceLogData.java c6fdba3 Diff: https://reviews.apache.org/r/64083/diff/3/ Changes: https://reviews.apache.org/r/64083/diff/2-3/ Testing --- done. Thanks, Oliver Szabo
Re: Review Request 64083: Log Search: Adding new dynamic field for service logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64083/ --- (Updated Nov. 27, 2017, 2:54 p.m.) Review request for Ambari, Attila Doroszlai, Krisztian Kasa, and Robert Nettleton. Changes --- fix dynamic field name Bugs: Ambari https://issues.apache.org/jira/browse/Ambari Repository: ambari Description --- its required to add in order to have a dynamic field which not only indexed, but stored as well. Diffs (updated) - ambari-logsearch/ambari-logsearch-server/src/main/configsets/hadoop_logs/conf/managed-schema c6f498b ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/SolrConstants.java 60fc1a3 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/model/SolrCommonLogData.java b5ce7ad ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/model/SolrServiceLogData.java c6fdba3 Diff: https://reviews.apache.org/r/64083/diff/2/ Changes: https://reviews.apache.org/r/64083/diff/1-2/ Testing --- done. Thanks, Oliver Szabo
Review Request 64083: Log Search: Adding new dynamic field for service logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64083/ --- Review request for Ambari, Attila Doroszlai and Krisztian Kasa. Bugs: Ambari https://issues.apache.org/jira/browse/Ambari Repository: ambari Description --- its required to add in order to have a dynamic field which not only indexed, but stored as well. Diffs - ambari-logsearch/ambari-logsearch-server/src/main/configsets/hadoop_logs/conf/managed-schema c6f498b ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/SolrConstants.java 60fc1a3 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/model/SolrServiceLogData.java c6fdba3 Diff: https://reviews.apache.org/r/64083/diff/1/ Testing --- done. Thanks, Oliver Szabo
Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64059/#review191833 --- ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java Lines 27 (patched) <https://reviews.apache.org/r/64059/#comment269774> it can be service log source ambari-infra/pom.xml Lines 163 (patched) <https://reviews.apache.org/r/64059/#comment269773> is this really needed? is it becase of you reached the limit of the log file size for long running and it started to rotate? also it is too specific so it can be * * / * . log - * (i just use spaces to the editor not format it) - Oliver Szabo On Nov. 24, 2017, 9:31 a.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64059/ > --- > > (Updated Nov. 24, 2017, 9:31 a.m.) > > > Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton. > > > Bugs: AMBARI-22514 > https://issues.apache.org/jira/browse/AMBARI-22514 > > > Repository: ambari > > > Description > --- > > Initial implementation of Schedulable document deletion & archiving for Infra > Solr > == > > Spring bacth job for > > * exporting solr documents from hadoop_logs collection > * saving the documents to filesystem. All document is converted to json and > written into new line. > * compress each file using tar.gz > * job can be started using ambari-infra-manager api and parameters can be > overwritten > * s3 upload support (not used yet) > > > Diffs > - > > ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java > PRE-CREATION > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manage
Review Request 63832: Reference Log Feeder IDE/maven dev setup in README.md
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63832/ --- Review request for Ambari, Attila Doroszlai, Attila Magyar, Krisztian Kasa, and Robert Nettleton. Bugs: AMBARI-22450 https://issues.apache.org/jira/browse/AMBARI-22450 Repository: ambari Description --- small patch to reference logfeeder reamdme in the parent readme Diffs - ambari-logsearch/README.md 33952a2 Diff: https://reviews.apache.org/r/63832/diff/1/ Testing --- not required. Thanks, Oliver Szabo
Review Request 63803: Log Feeder: do not try to connect AMS, when it is disabled
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63803/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22443 https://issues.apache.org/jira/browse/AMBARI-22443 Repository: ambari Description --- Set ams client to null if any of their properties are null. Its already merged/fixed in 2.6 branch Diffs - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java f446446 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java 1094852 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetricsManagerTest.java 24042a7 Diff: https://reviews.apache.org/r/63803/diff/1/ Testing --- done Thanks, Oliver Szabo
Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/ --- (Updated Nov. 14, 2017, 1:07 p.m.) Review request for Ambari, Krisztian Kasa and Robert Nettleton. Changes --- use waitIntervalMin as parameter Bugs: AMBARI-22395 https://issues.apache.org/jira/browse/AMBARI-22395 Repository: ambari Description --- - added new config: "max_age_min" (similar as detach_time_min) to input block. that stored in checkpoint files, during cleanup, based on that the checkpoint file can be deleted. - based on "max_age_min" value, during bootstrap, its possible logfeeder wont even monitor a file in the future. - created a new thread which periodically run the checkpoint cleanup (now in every 2000 minutes ... its ok to run that rearly) - fix metrics sink, as its called if some properties are missing - add new input test files (testing wildcard pattern) - fix when fatal error thrown if no any files exist based on a wildcard pattern Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 8548a20 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 96320e9 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8e70850 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java 3910b9b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java 322a56d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java cc5d664 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java 2d1bf40 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java 1432c87 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java 843ae6b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java 8ee6d00 ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json PRE-CREATION ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log PRE-CREATION Diff: https://reviews.apache.org/r/63793/diff/3/ Changes: https://reviews.apache.org/r/63793/diff/2-3/ Testing --- done, manually, added new input file for docker env. Thanks, Oliver Szabo
Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/ --- (Updated Nov. 14, 2017, 12:50 p.m.) Review request for Ambari, Krisztian Kasa and Robert Nettleton. Changes --- handle that if maxAgeMin value set to zero Bugs: AMBARI-22395 https://issues.apache.org/jira/browse/AMBARI-22395 Repository: ambari Description --- - added new config: "max_age_min" (similar as detach_time_min) to input block. that stored in checkpoint files, during cleanup, based on that the checkpoint file can be deleted. - based on "max_age_min" value, during bootstrap, its possible logfeeder wont even monitor a file in the future. - created a new thread which periodically run the checkpoint cleanup (now in every 2000 minutes ... its ok to run that rearly) - fix metrics sink, as its called if some properties are missing - add new input test files (testing wildcard pattern) - fix when fatal error thrown if no any files exist based on a wildcard pattern Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 8548a20 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 96320e9 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8e70850 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java 3910b9b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java 322a56d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java cc5d664 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java 2d1bf40 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java 1432c87 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java 843ae6b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java 8ee6d00 ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json PRE-CREATION ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log PRE-CREATION Diff: https://reviews.apache.org/r/63793/diff/2/ Changes: https://reviews.apache.org/r/63793/diff/1-2/ Testing --- done, manually, added new input file for docker env. Thanks, Oliver Szabo
Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/ --- Review request for Ambari, Krisztian Kasa and Robert Nettleton. Bugs: AMBARI-22395 https://issues.apache.org/jira/browse/AMBARI-22395 Repository: ambari Description --- - added new config: "max_age_min" (similar as detach_time_min) to input block. that stored in checkpoint files, during cleanup, based on that the checkpoint file can be deleted. - based on "max_age_min" value, during bootstrap, its possible logfeeder wont even monitor a file in the future. - created a new thread which periodically run the checkpoint cleanup (now in every 2000 minutes ... its ok to run that rearly) - fix metrics sink, as its called if some properties are missing - add new input test files (testing wildcard pattern) - fix when fatal error thrown if no any files exist based on a wildcard pattern Diffs - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 8548a20 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 96320e9 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8e70850 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java 3910b9b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java 322a56d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java cc5d664 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java 2d1bf40 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java 1432c87 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java 843ae6b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java 8ee6d00 ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json PRE-CREATION ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log PRE-CREATION Diff: https://reviews.apache.org/r/63793/diff/1/ Testing --- done, manually, added new input file for docker env. Thanks, Oliver Szabo
Re: Review Request 63696: Exclude test scope jars from Log Search / Log Feeder rpm/deb packages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63696/ --- (Updated Nov. 9, 2017, 1:21 p.m.) Review request for Ambari, Attila Doroszlai, Attila Magyar, Aravindan Vijayan, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Changes --- swap branches + bug Bugs: AMBARI-22389 https://issues.apache.org/jira/browse/AMBARI-22389 Repository: ambari Description --- dependency plugin include all jars in target/libs, so we need to remove those that has test scope Diffs - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml cf64a36 ambari-logsearch/ambari-logsearch-portal/pom.xml 53d6e8b Diff: https://reviews.apache.org/r/63696/diff/2/ Testing --- UT done, target/libs folder does not contain easymock, checkstyle, utility,junit jars Thanks, Oliver Szabo
Re: Review Request 63696: Exclude test scope jars from Log Search / Log Feeder rpm/deb packages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63696/ --- (Updated Nov. 9, 2017, 12:53 p.m.) Review request for Ambari, Attila Doroszlai, Attila Magyar, Aravindan Vijayan, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Changes --- remove artifactItems Bugs: branch-2.6 and trunk https://issues.apache.org/jira/browse/branch-2.6 https://issues.apache.org/jira/browse/trunk Repository: ambari Description --- dependency plugin include all jars in target/libs, so we need to remove those that has test scope Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml cf64a36 ambari-logsearch/ambari-logsearch-portal/pom.xml 53d6e8b Diff: https://reviews.apache.org/r/63696/diff/2/ Changes: https://reviews.apache.org/r/63696/diff/1-2/ Testing --- UT done, target/libs folder does not contain easymock, checkstyle, utility,junit jars Thanks, Oliver Szabo
Review Request 63696: Exclude test scope jars from Log Search / Log Feeder rpm/deb packages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63696/ --- Review request for Ambari, Attila Doroszlai, Attila Magyar, Aravindan Vijayan, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: branch-2.6 and trunk https://issues.apache.org/jira/browse/branch-2.6 https://issues.apache.org/jira/browse/trunk Repository: ambari Description --- dependency plugin include all jars in target/libs, so we need to remove those that has test scope Diffs - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml cf64a36 ambari-logsearch/ambari-logsearch-portal/pom.xml 53d6e8b Diff: https://reviews.apache.org/r/63696/diff/1/ Testing --- UT done, target/libs folder does not contain easymock, checkstyle, utility,junit jars Thanks, Oliver Szabo
Review Request 63617: Log Feeder should be able to run locally with dockerized dev env by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63617/ --- Review request for Ambari, Krisztian Kasa and Robert Nettleton. Bugs: AMBARI-22364 https://issues.apache.org/jira/browse/AMBARI-22364 Repository: ambari Description --- Run logfeeder locally, requirements: - some changes in run logsearch configs (add --monitor flag, set workspace folder) logfeeder.properties work with docker environment Diffs - ambari-logsearch/ambari-logsearch-logfeeder/README.md 5a64daf ambari-logsearch/ambari-logsearch-logfeeder/pom.xml b1b6ece ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log-samples/.gitignore PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log-samples/logs/service_sample.txt PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log-samples/shipper-conf/global.config.json PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log-samples/shipper-conf/input.config-sample.json PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log-samples/shipper-conf/output.config-sample.json PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log4j.xml 8d5174e ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties 74ea0ef Diff: https://reviews.apache.org/r/63617/diff/1/ Testing --- done with docker-compose commands Thanks, Oliver Szabo
Re: Review Request 63525: Log Search server should be able to run locally with dockerized dev env by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63525/ --- (Updated Nov. 2, 2017, 9:35 p.m.) Review request for Ambari, Krisztian Kasa and Robert Nettleton. Changes --- fix markdown Bugs: AMBARI-22357 https://issues.apache.org/jira/browse/AMBARI-22357 Repository: ambari Description --- change defaults configs + add more proper descriptions to use / setup log search server local dev env quickly (with maven / IDE) Diffs (updated) - ambari-logsearch/README.md 43878f2 ambari-logsearch/ambari-logsearch-server/.gitignore 07e0389 ambari-logsearch/ambari-logsearch-server/README.md 126f651 ambari-logsearch/ambari-logsearch-server/run.sh 765fe6c ambari-logsearch/ambari-logsearch-server/src/main/resources/logsearch.properties db8a0a2 Diff: https://reviews.apache.org/r/63525/diff/2/ Changes: https://reviews.apache.org/r/63525/diff/1-2/ Testing --- done, check with 'mvn clean spring-boot:run' and with IntelliJ as well. Thanks, Oliver Szabo
Review Request 63525: Log Search server should be able to run locally with dockerized dev env by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63525/ --- Review request for Ambari, Krisztian Kasa and Robert Nettleton. Bugs: AMBARI-22357 https://issues.apache.org/jira/browse/AMBARI-22357 Repository: ambari Description --- change defaults configs + add more proper descriptions to use / setup log search server local dev env quickly (with maven / IDE) Diffs - ambari-logsearch/README.md 43878f2 ambari-logsearch/ambari-logsearch-server/.gitignore 07e0389 ambari-logsearch/ambari-logsearch-server/README.md 126f651 ambari-logsearch/ambari-logsearch-server/run.sh 765fe6c ambari-logsearch/ambari-logsearch-server/src/main/resources/logsearch.properties db8a0a2 Diff: https://reviews.apache.org/r/63525/diff/1/ Testing --- done, check with 'mvn clean spring-boot:run' and with IntelliJ as well. Thanks, Oliver Szabo
Review Request 63429: Log Search: add descriptions about how to use Log Search environment with docker compose
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63429/ --- Review request for Ambari, Krisztian Kasa and Robert Nettleton. Bugs: AMBARI-22331 https://issues.apache.org/jira/browse/AMBARI-22331 Repository: ambari Description --- Update README.md of Log Search Diffs - ambari-logsearch/README.md d05f45a Diff: https://reviews.apache.org/r/63429/diff/1/ Testing --- not required Thanks, Oliver Szabo
Re: Review Request 63416: Remove python & npm install from Log Search docker container
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63416/#review189600 --- Ship it! Ship It! - Oliver Szabo On Oct. 30, 2017, 3:47 p.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63416/ > --- > > (Updated Oct. 30, 2017, 3:47 p.m.) > > > Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sandor Magyari. > > > Bugs: AMBARI-22334 > https://issues.apache.org/jira/browse/AMBARI-22334 > > > Repository: ambari > > > Description > --- > > Remove python & npm install from Log Search docker container > Additional solr host fixed in order to access solr from zookeper > > > Diffs > - > > ambari-logsearch/docker/Dockerfile 70e80b65e9 > ambari-logsearch/docker/docker-compose.yml a6eeb0df6f > ambari-logsearch/docker/logsearch-docker.sh b44e7ee43b > > > Diff: https://reviews.apache.org/r/63416/diff/1/ > > > Testing > --- > > manully with docker-compose > > > Thanks, > > Krisztian Kasa > >
Re: Review Request 63393: Log Search: Add docker compose file for Log Search server and Log Feeder
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63393/ --- (Updated Oct. 29, 2017, 10:19 a.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, Robert Nettleton, and Sebastian Toader. Changes --- fix zookeeper connection string & rat exclude for .env file Bugs: AMBARI-22329 https://issues.apache.org/jira/browse/AMBARI-22329 Repository: ambari Description --- Add docker compose files for logsearch & logfeeder. - add all.yml which contains all services in one container - logsearch-server.yml and logsearch-logfeeder.yml -> define the services separately - some updates in the logearch-docker script - create .env file for docker compose - merge solr662.yml and solr.yml files as .env file was added. - add docker-compose.yml as well which contains all services (by default that can be used, then with the specific ones, like solr.yml, you can use them to scale specific components - it can be used scale as those not expose any ports) - upgrade maven version in Dockerfile Diffs (updated) - ambari-logsearch/docker/.gitignore 99455ed ambari-logsearch/docker/Dockerfile 5906a50 ambari-logsearch/docker/all.yml PRE-CREATION ambari-logsearch/docker/bin/start.sh 883ba02 ambari-logsearch/docker/docker-compose.yml PRE-CREATION ambari-logsearch/docker/logsearch-docker.sh 1660cee ambari-logsearch/docker/logsearch-logfeeder.yml PRE-CREATION ambari-logsearch/docker/logsearch-server.yml PRE-CREATION ambari-logsearch/docker/solr.yml 727aba1 ambari-logsearch/docker/solr662.yml b0641ed ambari-logsearch/pom.xml 18ff45e Diff: https://reviews.apache.org/r/63393/diff/3/ Changes: https://reviews.apache.org/r/63393/diff/2-3/ Testing --- done with docker-compose commands Thanks, Oliver Szabo
Re: Review Request 63393: Log Search: Add docker compose file for Log Search server and Log Feeder
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63393/ --- (Updated Oct. 28, 2017, 10:49 p.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, Robert Nettleton, and Sebastian Toader. Changes --- add docker-compose.yml and use only one zookeeper in that. Bugs: AMBARI-22329 https://issues.apache.org/jira/browse/AMBARI-22329 Repository: ambari Description (updated) --- Add docker compose files for logsearch & logfeeder. - add all.yml which contains all services in one container - logsearch-server.yml and logsearch-logfeeder.yml -> define the services separately - some updates in the logearch-docker script - create .env file for docker compose - merge solr662.yml and solr.yml files as .env file was added. - add docker-compose.yml as well which contains all services (by default that can be used, then with the specific ones, like solr.yml, you can use them to scale specific components - it can be used scale as those not expose any ports) - upgrade maven version in Dockerfile Diffs (updated) - ambari-logsearch/docker/.gitignore 99455ed ambari-logsearch/docker/Dockerfile 5906a50 ambari-logsearch/docker/all.yml PRE-CREATION ambari-logsearch/docker/bin/start.sh 883ba02 ambari-logsearch/docker/docker-compose.yml PRE-CREATION ambari-logsearch/docker/logsearch-docker.sh 1660cee ambari-logsearch/docker/logsearch-logfeeder.yml PRE-CREATION ambari-logsearch/docker/logsearch-server.yml PRE-CREATION ambari-logsearch/docker/solr.yml 727aba1 ambari-logsearch/docker/solr662.yml b0641ed Diff: https://reviews.apache.org/r/63393/diff/2/ Changes: https://reviews.apache.org/r/63393/diff/1-2/ Testing --- done with docker-compose commands Thanks, Oliver Szabo
Review Request 63393: Log Search: Add docker compose file for Log Search server and Log Feeder
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63393/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, Robert Nettleton, and Sebastian Toader. Bugs: AMBARI-22329 https://issues.apache.org/jira/browse/AMBARI-22329 Repository: ambari Description --- Add docker compose files for logsearch & logfeeder. - add all.yml which contains all services in one container - logsearch-server.yml and logsearch-logfeeder.yml -> define the services separately - some updates in the logearch-docker script - create .env file for docker compose - merge solr662.yml and solr.yml files as .env file was added. Diffs - ambari-logsearch/docker/.gitignore 99455ed ambari-logsearch/docker/Dockerfile 5906a50 ambari-logsearch/docker/all.yml PRE-CREATION ambari-logsearch/docker/bin/start.sh 883ba02 ambari-logsearch/docker/logsearch-docker.sh 1660cee ambari-logsearch/docker/logsearch-logfeeder.yml PRE-CREATION ambari-logsearch/docker/logsearch-server.yml PRE-CREATION ambari-logsearch/docker/solr.yml 727aba1 ambari-logsearch/docker/solr662.yml b0641ed Diff: https://reviews.apache.org/r/63393/diff/1/ Testing --- done with docker-compose commands Thanks, Oliver Szabo
Re: Review Request 63362: Fix Log Search configuration dependencies
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63362/#review189500 --- Ship it! Ship It! - Oliver Szabo On Oct. 27, 2017, 4:28 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63362/ > --- > > (Updated Oct. 27, 2017, 4:28 p.m.) > > > Review request for Ambari, Krisztian Kasa, Oliver Szabo, and Robert Nettleton. > > > Bugs: AMBARI-22328 > https://issues.apache.org/jira/browse/AMBARI-22328 > > > Repository: ambari > > > Description > --- > > logsearch-service_logs-solrconfig and logsearch-audit_logs-solrconfig config > groups are specified with .xml, logsearch-common-properties is not present at > all as a dependency. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/metainfo.xml > 4a7b3e0 > > > Diff: https://reviews.apache.org/r/63362/diff/1/ > > > Testing > --- > > Tested on local cluster. > > > Thanks, > > Miklos Gergely > >
Re: Review Request 63360: Log Search: Add docker compose file for Solr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63360/ --- (Updated Oct. 27, 2017, 11:59 p.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Changes --- add License headers Bugs: AMBARI-22315 https://issues.apache.org/jira/browse/AMBARI-22315 Repository: ambari Description --- its the next part of https://reviews.apache.org/r/63328/ contains some renames as well Diffs (updated) - ambari-logsearch/docker/docker-compose-zookeeper.yml a41eaf3 ambari-logsearch/docker/solr.yml PRE-CREATION ambari-logsearch/docker/solr662.yml PRE-CREATION ambari-logsearch/docker/zookeeper.yml PRE-CREATION Diff: https://reviews.apache.org/r/63360/diff/2/ Changes: https://reviews.apache.org/r/63360/diff/1-2/ Testing --- done with docker-compose up -d Thanks, Oliver Szabo
Review Request 63360: Log Search: Add docker compose file for Solr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63360/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22315 https://issues.apache.org/jira/browse/AMBARI-22315 Repository: ambari Description --- its the next part of https://reviews.apache.org/r/63328/ contains some renames as well Diffs - ambari-logsearch/docker/docker-compose-zookeeper.yml a41eaf3 ambari-logsearch/docker/solr.yml PRE-CREATION ambari-logsearch/docker/solr662.yml PRE-CREATION ambari-logsearch/docker/zookeeper.yml PRE-CREATION Diff: https://reviews.apache.org/r/63360/diff/1/ Testing --- done with docker-compose up -d Thanks, Oliver Szabo
Review Request 63328: Log Search: Add docker-compose file for creating ZooKeeper containers
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63328/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22314 https://issues.apache.org/jira/browse/AMBARI-22314 Repository: ambari Description --- In order to separate components for logsearch dev env we need to create separate docker containers for all of them (zookeeper, solr, logsearch, logfeeder) Its much more clear to use a docker-compose file for it, and also we should use the offical zookeeper/solr images. First patch contains the zookeeper compose file. Diffs - ambari-logsearch/docker/docker-compose-zookeeper.yml PRE-CREATION Diff: https://reviews.apache.org/r/63328/diff/1/ Testing --- done, with "docker-compose -f docker-compose-zookeeper.yml up -d" Thanks, Oliver Szabo
Re: Review Request 63323: Get rid of deprecated Solr usages (infra + logsearch)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63323/ --- (Updated Oct. 26, 2017, 10:34 a.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Changes --- fix information panel Bugs: AMBARI-22310 https://issues.apache.org/jira/browse/AMBARI-22310 Repository: ambari Description --- As a lot of deprecated solr4j (solr 6+) methods will be deleted in solr 7 I get rid of most of the solr 6 usages. One thign what i missing is the HttpClientUtil kerberos configurer changes, about that, there is no class that can be compatible with both solr versions Diffs - ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/AmbariSolrCloudClientBuilder.java 54c4ae3 ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/commands/CreateCollectionCommand.java 53e96fd ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/commands/CreateShardCommand.java 9b57827 ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/commands/GetShardsCommand.java e49bfec Diff: https://reviews.apache.org/r/63323/diff/1/ Testing --- done. Thanks, Oliver Szabo
Review Request 63323: Get rid of deprecated Solr usages (infra + logsearch)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63323/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: trunk https://issues.apache.org/jira/browse/trunk Repository: ambari Description --- As a lot of deprecated solr4j (solr 6+) methods will be deleted in solr 7 I get rid of most of the solr 6 usages. One thign what i missing is the HttpClientUtil kerberos configurer changes, about that, there is no class that can be compatible with both solr versions Diffs - ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/AmbariSolrCloudClientBuilder.java 54c4ae3 ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/commands/CreateCollectionCommand.java 53e96fd ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/commands/CreateShardCommand.java 9b57827 ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/commands/GetShardsCommand.java e49bfec Diff: https://reviews.apache.org/r/63323/diff/1/ Testing --- done. Thanks, Oliver Szabo
Review Request 63298: Update Solr (Infra Solr + clients) version to 6.6.2 from 6.6.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63298/ --- Review request for Ambari, Laszlo Puskas, Miklos Gergely, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-22305 https://issues.apache.org/jira/browse/AMBARI-22305 Repository: ambari Description --- Update Solr version to 6.6.2 from 6.6.0 (contains some security fixes) Diffs - ambari-infra/pom.xml 908abb4 ambari-logsearch/docker/Dockerfile 1e4135e ambari-logsearch/pom.xml 9434365 Diff: https://reviews.apache.org/r/63298/diff/1/ Testing --- done. Thanks, Oliver Szabo
Review Request 63265: Log Feeder: allow applying grok filter on default fields (like type, path, host etc.)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63265/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22300 https://issues.apache.org/jira/browse/AMBARI-22300 Repository: ambari Description --- add new "init_default_fields" config param to input block. If that option is turned on, then logfeeder will apply the default fields before any grok filter is applied (right now it should just work with grok filter Diffs - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java deff1b2 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 6f6bb6d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java 8bf6ae3 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java 73cf449 Diff: https://reviews.apache.org/r/63265/diff/1/ Testing --- UTs done Thanks, Oliver Szabo
Re: Review Request 63196: Avoid Log Feeder NPEs for logs of services not present on the node, but present on the cluster
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63196/#review188895 --- Ship it! Ship It! - Oliver Szabo On Oct. 21, 2017, 12:58 a.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63196/ > --- > > (Updated Oct. 21, 2017, 12:58 a.m.) > > > Review request for Ambari, Krisztian Kasa, Oliver Szabo, and Robert Nettleton. > > > Bugs: AMBARI-22281 > https://issues.apache.org/jira/browse/AMBARI-22281 > > > Repository: ambari > > > Description > --- > > Due to a bug Log Feeders throw NPEs while searching for logs of services > which are not present on the node, but present on the cluster. > > The problem was caused by the fact that File.listFiles(FileFilter) returns > null if the file it's called on is not an existing directory, and > Arrays.sort() throws an NPE if it's argument is null. > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java > e24a7aa > > > Diff: https://reviews.apache.org/r/63196/diff/1/ > > > Testing > --- > > Tested on local cluster > > > Thanks, > > Miklos Gergely > >
Re: Review Request 63027: Allow wildcard for log directory folder in the path component of Logfeeder input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63027/ --- (Updated Oct. 20, 2017, 1:46 p.m.) Review request for Ambari, Miklos Gergely and Robert Nettleton. Changes --- threads not stopped properly Bugs: AMBARI-21145 https://issues.apache.org/jira/browse/AMBARI-21145 Repository: ambari Description --- Allow wildcards for handling log directories. Using ant library for this. (on trunk, the solution will be JDK 8 based, so its a temporal hack for 2.6.1, for trunk some classes have cleaned up, so the implementation there will be different) The idea is if we have a folder pattern, we will craete a thread group, and there will be different threads for every matching input (now only for tail file) Also there is 2 new thread introduced: - one which checks is the monitored file too old, if it is, stop the thread for that (detach) - an another one which checks there is any new file available (like log files with date patterns in its name), and if it is, update the monitoring thread with that (log path upgrade) Introduced some new configuration as well for inputs: - path_update_interval_min: the period in minutes for checking new files (default: 5, based on detach values, its possible that a new input wont be monitored) - detach_interval_min: the period in minutes for checking which files are too old (default: 300) - detach_time_min: the period in minutes when we flag a file is too old (default: 2000) Also fix an issue which cause that we cannot use grok field names more deeply (use deep_extract param for this one in grok filter) and i updated the docker container as well to make it work with branch-2.6 (and use different version for container in order to keep that) Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 49122e8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 7e2da70 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 41a1fa5 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 9f54d8a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java 3737839 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandler.java 0ece637 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/docker/Dockerfile dfa1462 ambari-logsearch/docker/logsearch-docker.sh a2df90f Diff: https://reviews.apache.org/r/63027/diff/6/ Changes: https://reviews.apache.org/r/63027/diff/5-6/ Testing --- done, no UTs here, FTs are in progress Thanks, Oliver Szabo
Re: Review Request 63162: Disable xmlparser and configEdit API in Infra Solr by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63162/ --- (Updated Oct. 19, 2017, 7:36 p.m.) Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Changes --- add missing SOLR_OPT value Bugs: AMBARI-22273 https://issues.apache.org/jira/browse/AMBARI-22273 Repository: ambari Description --- Due to a vulnerability found in Solr with CVE-2017-12629 (https://nvd.nist.gov/vuln/detail/CVE-2017-12629) 1.) Disable editing with the Config API by adding the "-Ddisable.configEdit=true" flag to the SOLR_OPTS by default. 2.) Update all collections to reroute the xmlparser query parser away from the vulnerable class, but adding this to the Ranger, Atlas, and LogSearch collections: That wont affect upgrade as with some manual changes these options can be set properly. This change only for default deployments. (also wont affect 3.0) Diffs (updated) - ambari-logsearch/ambari-logsearch-portal/src/main/configsets/audit_logs/conf/solrconfig.xml 7af91df ambari-logsearch/ambari-logsearch-portal/src/main/configsets/hadoop_logs/conf/solrconfig.xml 59f778f ambari-logsearch/ambari-logsearch-portal/src/main/configsets/history/conf/solrconfig.xml 8244a08 ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/infra-solr-env.sh.j2 5cc344e ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/atlas-solrconfig.xml cba4a4e ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/audit_logs-solrconfig.xml.j2 63879e7 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/service_logs-solrconfig.xml.j2 b6a4d1d ambari-server/src/main/resources/common-services/RANGER/0.7.0/properties/ranger-solrconfig.xml.j2 25dbb7a Diff: https://reviews.apache.org/r/63162/diff/2/ Changes: https://reviews.apache.org/r/63162/diff/1-2/ Testing --- done, UTs passes, FT: install Solr with these setting, also check what happens if we adding the new xml parser. Thanks, Oliver Szabo
Review Request 63162: Disable xmlparser and configEdit API in Infra Solr by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63162/ --- Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22273 https://issues.apache.org/jira/browse/AMBARI-22273 Repository: ambari Description --- Due to a vulnerability found in Solr with CVE-2017-12629 (https://nvd.nist.gov/vuln/detail/CVE-2017-12629) 1.) Disable editing with the Config API by adding the "-Ddisable.configEdit=true" flag to the SOLR_OPTS by default. 2.) Update all collections to reroute the xmlparser query parser away from the vulnerable class, but adding this to the Ranger, Atlas, and LogSearch collections: That wont affect upgrade as with some manual changes these options can be set properly. This change only for default deployments. (also wont affect 3.0) Diffs - ambari-logsearch/ambari-logsearch-portal/src/main/configsets/audit_logs/conf/solrconfig.xml 7af91df ambari-logsearch/ambari-logsearch-portal/src/main/configsets/hadoop_logs/conf/solrconfig.xml 59f778f ambari-logsearch/ambari-logsearch-portal/src/main/configsets/history/conf/solrconfig.xml 8244a08 ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/atlas-solrconfig.xml cba4a4e ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/audit_logs-solrconfig.xml.j2 63879e7 ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/service_logs-solrconfig.xml.j2 b6a4d1d ambari-server/src/main/resources/common-services/RANGER/0.7.0/properties/ranger-solrconfig.xml.j2 25dbb7a Diff: https://reviews.apache.org/r/63162/diff/1/ Testing --- done, UTs passes, FT: install Solr with these setting, also check what happens if we adding the new xml parser. Thanks, Oliver Szabo
Review Request 63118: Log Search server does not handle proxies properly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63118/ --- Review request for Ambari, Miklos Gergely and Robert Nettleton. Bugs: AMBARI-22266 https://issues.apache.org/jira/browse/AMBARI-22266 Repository: ambari Description --- If logsearch running behind a proxy and the address is not on root, UI cannot be accessed/handled properly. (like: localhost:61888 can be accessed through localhost:8080/logsearch/) Regarding to that issue, there are some backend changes that required: - form login filter needs to be removed, as we have are custom login entripoint already (that form login filter is basically the same, so we had 2 different filter on each other, which did the same) - add useForward (configurable) to our login entrypoint, to make it possible to send redirects through proxies properly - on logout: add useReferer, which can be used if logsearch is behind a proxy, it can use the referer location instead of the root one. Diffs - ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/conf/AuthPropsConfig.java 54cc10c ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/conf/SecurityConfig.java 2f9cba4 ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/web/authenticate/LogsearchLogoutSuccessHandler.java c20e383 ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/web/filters/LogsearchAuthenticationEntryPoint.java 1831697 Diff: https://reviews.apache.org/r/63118/diff/1/ Testing --- done, works with both proxies and without proxies (for proxies there should be some ui changes as well) Thanks, Oliver Szabo
Re: Review Request 63027: Allow wildcard for log directory folder in the path component of Logfeeder input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63027/ --- (Updated Oct. 18, 2017, 9:57 a.m.) Review request for Ambari, Miklos Gergely and Robert Nettleton. Changes --- set multi folder value on init Bugs: AMBARI-21145 https://issues.apache.org/jira/browse/AMBARI-21145 Repository: ambari Description --- Allow wildcards for handling log directories. Using ant library for this. (on trunk, the solution will be JDK 8 based, so its a temporal hack for 2.6.1, for trunk some classes have cleaned up, so the implementation there will be different) The idea is if we have a folder pattern, we will craete a thread group, and there will be different threads for every matching input (now only for tail file) Also there is 2 new thread introduced: - one which checks is the monitored file too old, if it is, stop the thread for that (detach) - an another one which checks there is any new file available (like log files with date patterns in its name), and if it is, update the monitoring thread with that (log path upgrade) Introduced some new configuration as well for inputs: - path_update_interval_min: the period in minutes for checking new files (default: 5, based on detach values, its possible that a new input wont be monitored) - detach_interval_min: the period in minutes for checking which files are too old (default: 300) - detach_time_min: the period in minutes when we flag a file is too old (default: 2000) Also fix an issue which cause that we cannot use grok field names more deeply (use deep_extract param for this one in grok filter) and i updated the docker container as well to make it work with branch-2.6 (and use different version for container in order to keep that) Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 49122e8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 7e2da70 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 41a1fa5 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 9f54d8a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java 3737839 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandler.java 0ece637 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/docker/Dockerfile dfa1462 ambari-logsearch/docker/logsearch-docker.sh a2df90f Diff: https://reviews.apache.org/r/63027/diff/5/ Changes: https://reviews.apache.org/r/63027/diff/4-5/ Testing --- done, no UTs here, FTs are in progress Thanks, Oliver Szabo
Re: Review Request 63027: Allow wildcard for log directory folder in the path component of Logfeeder input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63027/ --- (Updated Oct. 18, 2017, 9:38 a.m.) Review request for Ambari, Miklos Gergely and Robert Nettleton. Changes --- - getting regex log file path fix Bugs: AMBARI-21145 https://issues.apache.org/jira/browse/AMBARI-21145 Repository: ambari Description --- Allow wildcards for handling log directories. Using ant library for this. (on trunk, the solution will be JDK 8 based, so its a temporal hack for 2.6.1, for trunk some classes have cleaned up, so the implementation there will be different) The idea is if we have a folder pattern, we will craete a thread group, and there will be different threads for every matching input (now only for tail file) Also there is 2 new thread introduced: - one which checks is the monitored file too old, if it is, stop the thread for that (detach) - an another one which checks there is any new file available (like log files with date patterns in its name), and if it is, update the monitoring thread with that (log path upgrade) Introduced some new configuration as well for inputs: - path_update_interval_min: the period in minutes for checking new files (default: 5, based on detach values, its possible that a new input wont be monitored) - detach_interval_min: the period in minutes for checking which files are too old (default: 300) - detach_time_min: the period in minutes when we flag a file is too old (default: 2000) Also fix an issue which cause that we cannot use grok field names more deeply (use deep_extract param for this one in grok filter) and i updated the docker container as well to make it work with branch-2.6 (and use different version for container in order to keep that) Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 49122e8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 7e2da70 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 9f54d8a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java 3737839 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandler.java 0ece637 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/docker/Dockerfile dfa1462 ambari-logsearch/docker/logsearch-docker.sh a2df90f Diff: https://reviews.apache.org/r/63027/diff/4/ Changes: https://reviews.apache.org/r/63027/diff/3-4/ Testing --- done, no UTs here, FTs are in progress Thanks, Oliver Szabo
Re: Review Request 63027: Allow wildcard for log directory folder in the path component of Logfeeder input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63027/ --- (Updated Oct. 18, 2017, 9:24 a.m.) Review request for Ambari, Miklos Gergely and Robert Nettleton. Changes --- change FileUtils getInputFiles method name Bugs: AMBARI-21145 https://issues.apache.org/jira/browse/AMBARI-21145 Repository: ambari Description --- Allow wildcards for handling log directories. Using ant library for this. (on trunk, the solution will be JDK 8 based, so its a temporal hack for 2.6.1, for trunk some classes have cleaned up, so the implementation there will be different) The idea is if we have a folder pattern, we will craete a thread group, and there will be different threads for every matching input (now only for tail file) Also there is 2 new thread introduced: - one which checks is the monitored file too old, if it is, stop the thread for that (detach) - an another one which checks there is any new file available (like log files with date patterns in its name), and if it is, update the monitoring thread with that (log path upgrade) Introduced some new configuration as well for inputs: - path_update_interval_min: the period in minutes for checking new files (default: 5, based on detach values, its possible that a new input wont be monitored) - detach_interval_min: the period in minutes for checking which files are too old (default: 300) - detach_time_min: the period in minutes when we flag a file is too old (default: 2000) Also fix an issue which cause that we cannot use grok field names more deeply (use deep_extract param for this one in grok filter) and i updated the docker container as well to make it work with branch-2.6 (and use different version for container in order to keep that) Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 49122e8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 7e2da70 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 9f54d8a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java 3737839 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandler.java 0ece637 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/docker/Dockerfile dfa1462 ambari-logsearch/docker/logsearch-docker.sh a2df90f Diff: https://reviews.apache.org/r/63027/diff/3/ Changes: https://reviews.apache.org/r/63027/diff/2-3/ Testing --- done, no UTs here, FTs are in progress Thanks, Oliver Szabo
Re: Review Request 63027: Allow wildcard for log directory folder in the path component of Logfeeder input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63027/ --- (Updated Oct. 18, 2017, 9:19 a.m.) Review request for Ambari, Miklos Gergely and Robert Nettleton. Changes --- - too much wait for wait intervals - do some recommended changes by Miklos Bugs: AMBARI-21145 https://issues.apache.org/jira/browse/AMBARI-21145 Repository: ambari Description --- Allow wildcards for handling log directories. Using ant library for this. (on trunk, the solution will be JDK 8 based, so its a temporal hack for 2.6.1, for trunk some classes have cleaned up, so the implementation there will be different) The idea is if we have a folder pattern, we will craete a thread group, and there will be different threads for every matching input (now only for tail file) Also there is 2 new thread introduced: - one which checks is the monitored file too old, if it is, stop the thread for that (detach) - an another one which checks there is any new file available (like log files with date patterns in its name), and if it is, update the monitoring thread with that (log path upgrade) Introduced some new configuration as well for inputs: - path_update_interval_min: the period in minutes for checking new files (default: 5, based on detach values, its possible that a new input wont be monitored) - detach_interval_min: the period in minutes for checking which files are too old (default: 300) - detach_time_min: the period in minutes when we flag a file is too old (default: 2000) Also fix an issue which cause that we cannot use grok field names more deeply (use deep_extract param for this one in grok filter) and i updated the docker container as well to make it work with branch-2.6 (and use different version for container in order to keep that) Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 49122e8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 7e2da70 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 9f54d8a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java 3737839 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandler.java 0ece637 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/docker/Dockerfile dfa1462 ambari-logsearch/docker/logsearch-docker.sh a2df90f Diff: https://reviews.apache.org/r/63027/diff/2/ Changes: https://reviews.apache.org/r/63027/diff/1-2/ Testing --- done, no UTs here, FTs are in progress Thanks, Oliver Szabo
Review Request 63027: Allow wildcard for log directory folder in the path component of Logfeeder input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63027/ --- Review request for Ambari, Miklos Gergely and Robert Nettleton. Bugs: AMBARI-21145 https://issues.apache.org/jira/browse/AMBARI-21145 Repository: ambari Description --- Allow wildcards for handling log directories. Using ant library for this. (on trunk, the solution will be JDK 8 based, so its a temporal hack for 2.6.1, for trunk some classes have cleaned up, so the implementation there will be different) The idea is if we have a folder pattern, we will craete a thread group, and there will be different threads for every matching input (now only for tail file) Also there is 2 new thread introduced: - one which checks is the monitored file too old, if it is, stop the thread for that (detach) - an another one which checks there is any new file available (like log files with date patterns in its name), and if it is, update the monitoring thread with that (log path upgrade) Introduced some new configuration as well for inputs: - path_update_interval_min: the period in minutes for checking new files (default: 5, based on detach values, its possible that a new input wont be monitored) - detach_interval_min: the period in minutes for checking which files are too old (default: 300) - detach_time_min: the period in minutes when we flag a file is too old (default: 2000) Also fix an issue which cause that we cannot use grok field names more deeply (use deep_extract param for this one in grok filter) and i updated the docker container as well to make it work with branch-2.6 (and use different version for container in order to keep that) Diffs - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 49122e8 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java 7e2da70 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 9f54d8a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java 3737839 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandler.java 0ece637 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/docker/Dockerfile dfa1462 ambari-logsearch/docker/logsearch-docker.sh a2df90f Diff: https://reviews.apache.org/r/63027/diff/1/ Testing --- done, no UTs here, FTs are in progress Thanks, Oliver Szabo
Re: Review Request 62809: AMBARI-22159. Replace hostgroup vars for Druid
> On Oct. 7, 2017, noon, Oliver Szabo wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
> > Lines 2783 (patched)
> > <https://reviews.apache.org/r/62809/diff/1/?file=1847507#file1847507line2783>
> >
> > can these variables calculated at params.py in the stack code instead
> > of putting to the BlueprintConfigurationProcessor? (using jinja2 variables
> > at configurations - you can get the proper hosts from command.json input).
> > just because the ambari server java code could be independent from druid.
> > im not too big fan of this class, we should get rid of it sooner or later
> >
> > but you can drop this issue if zou do not want to do it, it's just a
> > reccomendation from me.
>
> Attila Doroszlai wrote:
> I agree that this class stinks. However, currently no `%HOSTGROUP::...%`
> substitution occurs in Python code. I guess it shouldn't be introduced at
> this point before 2.6 release.
no need for %HOSTGROUP::.. substitution. It can be a jinja variable like {{
metadata_connector_connect_uri }}.
like for logsearch and ambari infra solr, there are some refrerences for
zookeeper but you cannot find that in the BlueprintConfigurationProcessor class
- Oliver
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62809/#review187335
---
On Oct. 6, 2017, 3:35 p.m., Attila Doroszlai wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62809/
> ---
>
> (Updated Oct. 6, 2017, 3:35 p.m.)
>
>
> Review request for Ambari, Nishant Bangarwa, Slim Bouguerra, and Swapan
> Shridhar.
>
>
> Bugs: AMBARI-22159
> https://issues.apache.org/jira/browse/AMBARI-22159
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Replace placeholders in `metastore_hostname`,
> `druid.metadata.storage.connector.connectURI` and `druid.zk.service.host`
> when creating cluster via blueprint.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
> 5a6e2cc0de7dc0c0f4499624e21c43bbc8d24b54
>
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
> 68d6349db0207ab6df44b4b0a341c77ecabc46ea
>
>
> Diff: https://reviews.apache.org/r/62809/diff/1/
>
>
> Testing
> ---
>
> Tested manually on local cluster.
>
> Unit tests pass: `Tests run: 4900, Failures: 0, Errors: 0, Skipped: 36`
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 62809: AMBARI-22159. Replace hostgroup vars for Druid
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62809/#review187335 --- ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java Lines 2783 (patched) <https://reviews.apache.org/r/62809/#comment264257> can these variables calculated at params.py in the stack code instead of putting to the BlueprintConfigurationProcessor? (using jinja2 variables at configurations - you can get the proper hosts from command.json input). just because the ambari server java code could be independent from druid. im not too big fan of this class, we should get rid of it sooner or later but you can drop this issue if zou do not want to do it, it's just a reccomendation from me. - Oliver Szabo On Oct. 6, 2017, 3:35 p.m., Attila Doroszlai wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62809/ > --- > > (Updated Oct. 6, 2017, 3:35 p.m.) > > > Review request for Ambari, Nishant Bangarwa, Slim Bouguerra, and Swapan > Shridhar. > > > Bugs: AMBARI-22159 > https://issues.apache.org/jira/browse/AMBARI-22159 > > > Repository: ambari > > > Description > --- > > Replace placeholders in `metastore_hostname`, > `druid.metadata.storage.connector.connectURI` and `druid.zk.service.host` > when creating cluster via blueprint. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java > 5a6e2cc0de7dc0c0f4499624e21c43bbc8d24b54 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java > 68d6349db0207ab6df44b4b0a341c77ecabc46ea > > > Diff: https://reviews.apache.org/r/62809/diff/1/ > > > Testing > --- > > Tested manually on local cluster. > > Unit tests pass: `Tests run: 4900, Failures: 0, Errors: 0, Skipped: 36` > > > Thanks, > > Attila Doroszlai > >
Re: Review Request 62805: AMBARI-22155. Intermittent failure of FlumeTimelineMetricsSinkTest
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62805/#review187319 --- Ship it! Ship It! - Oliver Szabo On Oct. 6, 2017, 8:39 p.m., Attila Doroszlai wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62805/ > --- > > (Updated Oct. 6, 2017, 8:39 p.m.) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sid Wagle. > > > Bugs: AMBARI-22155 > https://issues.apache.org/jira/browse/AMBARI-22155 > > > Repository: ambari > > > Description > --- > > Mock the executor to eliminate nondeterminism caused by threads. > > > Diffs > - > > > ambari-metrics/ambari-metrics-flume-sink/src/main/java/org/apache/hadoop/metrics2/sink/flume/FlumeTimelineMetricsSink.java > 62779070419894064cb39ea5cc5da012572ff4cb > > ambari-metrics/ambari-metrics-flume-sink/src/test/java/org/apache/hadoop/metrics2/sink/flume/FlumeTimelineMetricsSinkTest.java > bd4ae6af88ee13f4ba6a3f14ac8c138b72d63c9d > > > Diff: https://reviews.apache.org/r/62805/diff/2/ > > > Testing > --- > > ``` > $ mvn -am -pl ambari-metrics/ambari-metrics-flume-sink clean test > ... > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 > ... > [INFO] Reactor Summary: > [INFO] > [INFO] Ambari Main SUCCESS [ 6.159 > s] > [INFO] Apache Ambari Project POM .. SUCCESS [ 0.036 > s] > [INFO] ambari-utility . SUCCESS [ 3.880 > s] > [INFO] ambari-metrics . SUCCESS [ 0.594 > s] > [INFO] Ambari Metrics Common .. SUCCESS [ 4.240 > s] > [INFO] Ambari Metrics Flume Sink .. SUCCESS [ 4.697 > s] > [INFO] > > [INFO] BUILD SUCCESS > ``` > > > Thanks, > > Attila Doroszlai > >
Re: Review Request 62723: AMBARI-22113. Unit test fails with Python 2.6
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62723/#review186862 --- Ship it! Ship It! - Oliver Szabo On Oct. 2, 2017, 6:12 p.m., Attila Doroszlai wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62723/ > --- > > (Updated Oct. 2, 2017, 6:12 p.m.) > > > Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sebastian Toader, > and Vitalyi Brodetskyi. > > > Bugs: AMBARI-22113 > https://issues.apache.org/jira/browse/AMBARI-22113 > > > Repository: ambari > > > Description > --- > > Eliminate/tweak Python 2.7+ constructs: > > * multiple contexts in `with` > * `assertIsNone` > * `assertRaises` > * `unittest.util` > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py > 86821bf47a757284f02ed5049fb29ccb71451340 > ambari-server/src/test/python/TestUpgradeSummary.py > 7606867f87f580e081c466ec889a6baa122b6c7a > > ambari-server/src/test/python/common-services/HIVE/test_jdbc_driver_config.py > e4d81b1bf650e15ac730c0e5d889ba2f98c32308 > > ambari-server/src/test/python/common-services/RANGER/test_db_flavor_config.py > 568e3fd7bb99ba888755382bce89065ea7d40914 > > ambari-server/src/test/python/common-services/RANGER_KMS/test_db_flavor_config.py > 48654ee18ad80aed35c3cab3d1ccbe6783be6b43 > > ambari-server/src/test/python/common-services/SQOOP/test_jdbc_driver_config.py > 7bb809a0a4e04565a1b201f08db8e0a93c3fd42c > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 0cc9ad0a603f5988e7f3af76babc9ad677c58149 > ambari-server/src/test/python/stacks/utils/RMFTestCase.py > 291f22b0fb1907cda4df4276aeff8a9df6d32b30 > > > Diff: https://reviews.apache.org/r/62723/diff/1/ > > > Testing > --- > > Tested with Python 2.6: > > ``` > $ mvn -am -pl ambari-server,ambari-agent -Drat.skip -Dcheckstyle.skip > -DskipSurefireTests clean test > ... > Total run:1202 > Total errors:0 > Total failures:0 > ... > Ran 454 tests in 22.430s > ... > [INFO] Ambari Server .. FAILURE [01:41 > min] > [INFO] Ambari Agent ... SUCCESS [ 28.188 > s] > ... > ``` > > and Python 2.7: > > ``` > $ mvn -am -pl ambari-server,ambari-agent -Drat.skip -Dcheckstyle.skip > -DskipSurefireTests clean test > ... > Total run:1202 > Total errors:0 > Total failures:0 > ... > Ran 457 tests in 16.645s > ... > [INFO] Ambari Server .. SUCCESS [01:35 > min] > [INFO] Ambari Agent ... SUCCESS [ 20.611 > s] > ``` > > > Thanks, > > Attila Doroszlai > >
Re: Review Request 62666: Upgrade Yarn version for Logsearch Web
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62666/#review186566 --- Ship it! Ship It! - Oliver Szabo On Sept. 28, 2017, 3:13 p.m., Istvan Tobias wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62666/ > --- > > (Updated Sept. 28, 2017, 3:13 p.m.) > > > Review request for Ambari, Andrii Babiichuk and Oliver Szabo. > > > Bugs: AMBARI-22079 > https://issues.apache.org/jira/browse/AMBARI-22079 > > > Repository: ambari > > > Description > --- > > Updating the package.json to use the latest node (8.6.0 - npm: 5.4.2) and > yarn (1.1.0). Upgraded maven pom file to use latest version of frontend > plugin. > > > Diffs > - > > ambari-logsearch/ambari-logsearch-web/package.json 7cf636f73a > ambari-logsearch/ambari-logsearch-web/pom.xml f27c2c6f7b > ambari-logsearch/ambari-logsearch-web/yarn.lock c55bb4d279 > > > Diff: https://reviews.apache.org/r/62666/diff/1/ > > > Testing > --- > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 01:31 min > [INFO] Finished at: 2017-09-28T17:06:10+02:00 > [INFO] Final Memory: 17M/58M > [INFO] > > > > Thanks, > > Istvan Tobias > >
Re: Review Request 62618: Use symlinks for Solr Data Manager
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62618/#review186412 --- Ship it! Ship It! - Oliver Szabo On Sept. 27, 2017, 2:33 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62618/ > --- > > (Updated Sept. 27, 2017, 2:33 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-22075 > https://issues.apache.org/jira/browse/AMBARI-22075 > > > Repository: ambari > > > Description > --- > > Put symlinks in /usr/bin for infra-solr-data-manager (same as > zookeeper-client and zookeeper-server), to acces > /usr/lib/ambari-infra-solr-client/solrDataManager.py more easily > > Create symlinks in the post install scripts, to make it possible to create > the rpm packages to be created on Mac > > > Diffs > - > > > ambari-infra/ambari-infra-assembly/src/main/package/deb/solr-client/postinst > c339cd0 > > ambari-infra/ambari-infra-assembly/src/main/package/rpm/solr-client/postinstall.sh > c339cd0 > ambari-infra/ambari-infra-solr-client/build.xml 25ff0cb > > > Diff: https://reviews.apache.org/r/62618/diff/1/ > > > Testing > --- > > tested on local cluster > > > Thanks, > > Miklos Gergely > >
Re: Review Request 62576: Solr Data Manager script should provide non-destructive archive download option
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62576/#review186389 --- Ship it! Shipt It! - Oliver Szabo On Sept. 26, 2017, 3:05 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62576/ > --- > > (Updated Sept. 26, 2017, 3:05 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-22061 > https://issues.apache.org/jira/browse/AMBARI-22061 > > > Repository: ambari > > > Description > --- > > Allow users to save only their data without deleting > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py > e0356bb > > > Diff: https://reviews.apache.org/r/62576/diff/1/ > > > Testing > --- > > tested locally > > > Thanks, > > Miklos Gergely > >
Review Request 62584: Remove Database auto configuration from Infra Manager
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62584/
---
Review request for Ambari, Attila Doroszlai, Laszlo Puskas, Miklos Gergely,
Robert Nettleton, and Sandor Magyari.
Bugs: AMBARI-22064
https://issues.apache.org/jira/browse/AMBARI-22064
Repository: ambari
Description
---
Spring boot introduced a Database autoconfiguration feature. Becuase of that
feature, our initialization script does not work.
To use that properly we need to set some spring properties ("spring.datasource"
prefix) which can use the right database schema script with discovering the
datasorce type of the datasource (which sounds fine), but we should not use
that feature until we find out how to change the default prefixes of the
configuration.
Diffs
-
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/InfraManager.java
185e344
Diff: https://reviews.apache.org/r/62584/diff/1/
Testing
---
manually, db file created properly
Thanks,
Oliver Szabo
Review Request 62570: Use symlinks for Infra components
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62570/ --- Review request for Ambari, Attila Doroszlai, Andrew Onischuk, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22059 https://issues.apache.org/jira/browse/AMBARI-22059 Repository: ambari Description --- Put symlinks in /usr/bin for infra-solr-client (same as zookeeper-client and zookeeper-server), to acces /usr/lib/ambari-infra-solr-client scripts more easily Create symlinks in the post install scripts, to make it possible to create the rpm packages to be created on Mac Diffs - ambari-infra/ambari-infra-assembly/pom.xml fafef7e ambari-infra/ambari-infra-assembly/src/main/package/deb/manager/postinst 21a01fa ambari-infra/ambari-infra-assembly/src/main/package/deb/solr-client/postinst 21a01fa ambari-infra/ambari-infra-assembly/src/main/package/rpm/manager/postinstall.sh PRE-CREATION ambari-infra/ambari-infra-assembly/src/main/package/rpm/solr-client/postinstall.sh PRE-CREATION ambari-infra/ambari-infra-manager/src/main/resources/infraManager.sh 65287b2 ambari-infra/ambari-infra-solr-client/src/main/resources/solrIndexHelper.sh 1d219d6 Diff: https://reviews.apache.org/r/62570/diff/1/ Testing --- done manually. Thanks, Oliver Szabo
Re: Review Request 62562: Solr Data Manager script should use gzip compression type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62562/#review186228 --- Ship it! Ship It! - Oliver Szabo On Sept. 26, 2017, 10:17 a.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62562/ > --- > > (Updated Sept. 26, 2017, 10:17 a.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-22056 > https://issues.apache.org/jira/browse/AMBARI-22056 > > > Repository: ambari > > > Description > --- > > Support gzip compression and offer it as default. > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py > 18a4da7 > > > Diff: https://reviews.apache.org/r/62562/diff/1/ > > > Testing > --- > > Tested on local cluster > > > Thanks, > > Miklos Gergely > >
Review Request 62551: Make Log Search IT work with the new UI
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62551/ --- Review request for Ambari, Attila Doroszlai, Miklos Gergely, Robert Nettleton, and Sandor Magyari. Bugs: AMBARI-22055 https://issues.apache.org/jira/browse/AMBARI-22055 Repository: ambari Description --- As Log Search has a new UI, integration test framework cannot login (with Selenium) anyomore during test run (using X11) Fix that to make IT tests work with the new UI Diffs - ambari-logsearch/ambari-logsearch-it/src/test/java/org/apache/ambari/logsearch/steps/LogSearchUISteps.java b40a2bc ambari-logsearch/ambari-logsearch-it/src/test/resources/stories/selenium/login.ui.story 543c211 Diff: https://reviews.apache.org/r/62551/diff/1/ Testing --- done. with maven failsafe plugin and enabling xquartz Thanks, Oliver Szabo
Re: Review Request 62542: Solr Data Manager should use --line-delimited option by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62542/#review186111 --- Ship it! Ship It! - Oliver Szabo On Sept. 25, 2017, 12:47 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62542/ > --- > > (Updated Sept. 25, 2017, 12:47 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-22049 > https://issues.apache.org/jira/browse/AMBARI-22049 > > > Repository: ambari > > > Description > --- > > Instead of having an option for creating line delimited json file, that > should be the default, and a json file should be created if a flag is set. > > Also the -j short argument form was assigned to the name parameter instead of > this. > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py > d17aec7 > > > Diff: https://reviews.apache.org/r/62542/diff/1/ > > > Testing > --- > > Tested on local cluster. > > > Thanks, > > Miklos Gergely > >
Re: Review Request 62543: Infra Manager jobs should handle their own configuration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62543/#review186110 --- Ship it! Ship It! - Oliver Szabo On Sept. 25, 2017, 12:52 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62543/ > --- > > (Updated Sept. 25, 2017, 12:52 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-22050 > https://issues.apache.org/jira/browse/AMBARI-22050 > > > Repository: ambari > > > Description > --- > > Dummy job configuration should be extracted from InfraManagerBatchConfig > > > Diffs > - > > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/conf/batch/InfraManagerBatchConfig.java > 98a4901 > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/dummy/DummyJobConfiguration.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/62543/diff/1/ > > > Testing > --- > > Tested locally, still works fine > > > Thanks, > > Miklos Gergely > >
Review Request 62530: Infra Manager: Integrate with Spring Boot
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62530/ --- Review request for Ambari, Attila Magyar, Laszlo Puskas, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22046 https://issues.apache.org/jira/browse/AMBARI-22046 Repository: ambari Description --- Integrate infra manager with spring boot in order to achive easier dependency and property management. Other improvements: - in memory swagger description storage - log4j2 support Diffs - ambari-infra/ambari-infra-manager/.gitignore PRE-CREATION ambari-infra/ambari-infra-manager/docker/bin/start.sh 076c06f ambari-infra/ambari-infra-manager/pom.xml b9f2a77 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/InfraManager.java 186f98c ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/common/InfraManagerConstants.java 105f20e ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/conf/InfraManagerDataConfig.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/conf/InfraManagerJerseyResourceConfig.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/conf/InfraManagerServletConfig.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/conf/StaticResourceConfiguration.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/doc/InfraManagerApiDocStorage.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/dummy/DummyItemWriter.java 444f0d1 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/rest/ApiDocResource.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties fbeac78 ambari-infra/ambari-infra-manager/src/main/resources/log4j.xml 0450454 ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml PRE-CREATION ambari-infra/ambari-infra-manager/src/main/resources/static/index.html PRE-CREATION ambari-infra/ambari-infra-manager/src/main/resources/swagger/swagger.html 8580e1a ambari-infra/ambari-infra-manager/src/main/resources/webapp/index.html 3e64867 Diff: https://reviews.apache.org/r/62530/diff/1/ Testing --- done manually. Thanks, Oliver Szabo
Review Request 62526: Infra Solr client cleanup: make infra-solr-client script usable with symlinks
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62526/ --- Review request for Ambari, Attila Magyar, Miklos Gergely, Robert Nettleton, and Sandor Magyari. Bugs: AMBARI-22045 https://issues.apache.org/jira/browse/AMBARI-22045 Repository: ambari Description --- Make it possible for infra solr scripts to used by symlinks. Right now if a symlink points to the infra solr scripts -> throws an error as we using the actual directory to calculate the java classpath. The patch fixes that. Also added some small cleanups (e.g.: stderrs are duplicated because of 2 appenders - i changed the default ones) Diffs - ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/AmbariSolrCloudCLI.java 70bc232 ambari-infra/ambari-infra-solr-client/src/main/resources/log4j.properties e8dca12 ambari-infra/ambari-infra-solr-client/src/main/resources/solrCloudCli.sh 7bfa864 ambari-infra/ambari-infra-solr-client/src/main/resources/solrIndexHelper.sh 12e6a77 ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/solr-client-log4j.properties.j2 0d0adc3 Diff: https://reviews.apache.org/r/62526/diff/1/ Testing --- tested manually. Thanks, Oliver Szabo
Review Request 62460: Enable logsearch-web unit tests and make them skippable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62460/ --- Review request for Ambari, Andrii Babiichuk, Attila Magyar, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-22019 https://issues.apache.org/jira/browse/AMBARI-22019 Repository: ambari Description --- The new Log Search web UTs are not enabled by default (as earlier it did not run in headless mode), now: enable it and make it skippable Diffs - ambari-logsearch/ambari-logsearch-web/pom.xml 7a6b02e Diff: https://reviews.apache.org/r/62460/diff/1/ Testing --- unit tests run and passed, skipped if i addeed -DskipTests Thanks, Oliver Szabo
Review Request 62456: Add JMX enable/disable option to Infra Solr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62456/ --- Review request for Ambari, Attila Magyar, Miklos Gergely, Robert Nettleton, and Sandor Magyari. Bugs: AMBARI-22013 https://issues.apache.org/jira/browse/AMBARI-22013 Repository: ambari Description --- - Add jmx enable option to Infra Solr stack (also change the default value from true to false) - Add java.rmi.server.hostname property to the solr env file (add this to the docker environment) to make JMX RMI communication work with JConsole. - Set the infra solr property ordering in ambari-web Diffs - ambari-logsearch/docker/test-config/solr/solr-env-ssl.sh faaa6ff ambari-logsearch/docker/test-config/solr/solr-env.sh 11b9fe0 ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-env.xml 52a5e6a ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py e4ea885 ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/infra-solr-env.sh.j2 5cc344e ambari-web/app/data/configs/services/ambari_infra_properties.js aae7fa5 Diff: https://reviews.apache.org/r/62456/diff/1/ Testing --- tested manually on 3 node cluster Thanks, Oliver Szabo
Review Request 62369: Log Search: remove file patterns from web security config
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62369/ --- Review request for Ambari, Andrii Babiichuk, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-21976 https://issues.apache.org/jira/browse/AMBARI-21976 Repository: ambari Description --- As the web resources files are mapped under /resources/, no need to map the file patterns in the security config Diffs - ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SecurityConfig.java 5ba0ac0 Diff: https://reviews.apache.org/r/62369/diff/1/ Testing --- done, manually Thanks, Oliver Szabo
Review Request 62368: Stack Advisor can reset the number of shards (Log Search) during adding a new service (BP install)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62368/ --- Review request for Ambari, Attila Doroszlai, Miklos Gergely, Robert Nettleton, and Sebastian Toader. Bugs: AMBARI-21975 https://issues.apache.org/jira/browse/AMBARI-21975 Repository: ambari Description --- The default value for the number of shards is 5. On UI install, the proper value is calculated, the max values is 3x number of solr instances. If you have only 1 Solr and Log Search was installed with BP without any custom shard config, the shard max value will be 3, but in fact the actual one is 5, that can cause during adding a new service through the UI can reset the number of shards. To fix that: set the max number of shard value to 5, in case you have just 1 solr ( use: max (3x solrs, 5)) Diffs - ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py f0d9f78 ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py ff57c7b Diff: https://reviews.apache.org/r/62368/diff/1/ Testing --- done. (updated unit tests) Thanks, Oliver Szabo
Review Request 62367: Increase the default maximum memory value for Infra Solr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62367/ --- Review request for Ambari, Attila Doroszlai, Miklos Gergely, Robert Nettleton, and Sebastian Toader. Bugs: AMBARI-21974 https://issues.apache.org/jira/browse/AMBARI-21974 Repository: ambari Description --- Change the max memory value for Infra Solr. The old max was 5G, which is sometimes too small for a solr node (there are examples where customers using 30G -> you cannot set that value easily on the UI with the smart configs) Diffs - ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-env.xml 27f001e Diff: https://reviews.apache.org/r/62367/diff/1/ Testing --- not required. Thanks, Oliver Szabo
Re: Review Request 62356: Log Feeder should not support german month names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62356/#review185482 --- Ship it! Ship It! - Oliver Szabo On Sept. 15, 2017, 1:27 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62356/ > --- > > (Updated Sept. 15, 2017, 1:27 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21968 > https://issues.apache.org/jira/browse/AMBARI-21968 > > > Repository: ambari > > > Description > --- > > Log Feeder grok patterns currently support german month names too. This has > almost no usefulness at all, as there are no german language logs, and we > don't even support german day names, but it may cause errors as the März > contains a ä character which may lead to encoding related errors. > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/grok-patterns > 920f521 > > > Diff: https://reviews.apache.org/r/62356/diff/1/ > > > Testing > --- > > Done. > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61898: Create Utility Script to support Solr Collection Data Retention/Purging/Archiving
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61898/#review185274 --- Ship it! Ship It! - Oliver Szabo On Sept. 12, 2017, 3:30 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61898/ > --- > > (Updated Sept. 12, 2017, 3:30 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21810 > https://issues.apache.org/jira/browse/AMBARI-21810 > > > Repository: ambari > > > Description > --- > > This script allows the user to set an end date, and save all the previous log > entries into HDFS or S3 > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/build.xml 9b8b6cc > ambari-infra/ambari-infra-solr-client/pom.xml 3818aba > > ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java > PRE-CREATION > ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61898/diff/7/ > > > Testing > --- > > Tested on local cluster with > - SSL > - kerberos > - interrupted, then restarted script > - HDFS > - S3 > - the combinations of the above > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61898: Create Utility Script to support Solr Collection Data Retention/Purging/Archiving
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61898/#review184886 --- ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py Lines 68 (patched) <https://reviews.apache.org/r/61898/#comment261085> this could be --line-delimited (by default would be the one json per line) - Oliver Szabo On Sept. 7, 2017, 8:33 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61898/ > --- > > (Updated Sept. 7, 2017, 8:33 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21810 > https://issues.apache.org/jira/browse/AMBARI-21810 > > > Repository: ambari > > > Description > --- > > This script allows the user to set an end date, and save all the previous log > entries into HDFS or S3 > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/build.xml 9b8b6cc > ambari-infra/ambari-infra-solr-client/pom.xml 3818aba > > ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java > PRE-CREATION > ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61898/diff/5/ > > > Testing > --- > > Tested on local cluster with > - SSL > - kerberos > - interrupted, then restarted script > - HDFS > - S3 > - the combinations of the above > > > Thanks, > > Miklos Gergely > >
Re: Review Request 62083: Fix Log Search upgrade to 3.0.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62083/#review184568 --- Ship it! Ship It! - Oliver Szabo On Sept. 5, 2017, 3:52 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62083/ > --- > > (Updated Sept. 5, 2017, 3:52 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21881 > https://issues.apache.org/jira/browse/AMBARI-21881 > > > Repository: ambari > > > Description > --- > > There are 3 problems about the Log Search upgrade to 3.0.0 > - *-logsearch-conf attributes were deleted instead of removing the config type > - logsearch-common-properties must be created explicitly, because the empty > config type will not be created by default > - logsearch-output-config was created instead of overwriting > logfeeder-output-config > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java > 9255163 > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog300Test.java > 53518dd > > > Diff: https://reviews.apache.org/r/62083/diff/1/ > > > Testing > --- > > Tested on local cluster. > > > Thanks, > > Miklos Gergely > >
Re: Review Request 62034: Integrate LogSearch new UI with the server and get rid of the old one
/ Testing --- done. Thanks, Oliver Szabo
Re: Review Request 62034: Integrate LogSearch new UI with the server and get rid of the old one
-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/Header_tmpl.html 24cc392 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/LogSnapShotLayout_tmpl.html 263df0c ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/TableLayout_tmpl.html 7b0c3eb ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/TimerView_tmpl.html e0960f7 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/TopNav_tmpl.html 2bbd349 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/breadcrumbs.html 26d5116 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/formInputItem.html b9e4b82 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/formInputItemList.html ff40e5b ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/common/loading_tmpl.html 2eb4fa8 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/BubbleGraphTableLayoutView_tmpl.html 4a2c88a ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/ComponentListView_tmpl.html 26a7782 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/ComponentsView_tmpl.html 383754e ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/DashboardView_tmpl.html dda24eb ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/GridTableLayoutView_tmpl.html 5feeb22 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/HostListView_tmpl.html 4af896e ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/HostsView_tmpl.html 58c680e ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/LogDetailView_tmpl.html c1a70f8 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/LogLevelBoxView_tmpl.html 0424cc0 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/LogLevelView_tmpl.html b790fb0 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dashboard/MainLayoutView_tmpl.html bc59f76 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dialog/ApplySearchFilterView_tmpl.html eacf5d0 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dialog/DetailLogFileView_tmpl.html dcd6e4a ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dialog/GlobalExclusionCompositeView_tmpl.html 1444fe0 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dialog/GlobalExclusionItemView_tmpl.html 2578f9b ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dialog/GridGraphSettingView_tmpl.html af311f3 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dialog/SaveSearchFilterView_tmpl.html fedce8e ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/dialog/TimeZoneChangeView_tmpl.html 1b5028e ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/filter/CreateLogfeederFilter_tmpl.html a4d78fa ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/graphs/GraphLayoutView_tmpl.html 02b384c ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/graphs/GridGraphLayoutView_tmpl.html 5feeb22 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/graphs/backup.js e250b02 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/helpers/Helpers.js 151335d ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/ComparisonLayoutView_tmpl.html 4de4878 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/ComparisonView_tmpl.html 0e7834e ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/EventHistoryLayoutView_tmpl.html add8482 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/ExportLogFileView_tmpl.html 2424c39 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/HierarchyTabLayoutView_tmpl.html d0d57ac ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/HostInfoTabLayoutView_tmpl.html a4c21e9 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/LogFileView_tmpl.html 0f0f143 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/TreeView_tmpl.html 3420ca2 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/tabs/VisualSearchView_tmpl.html 9043391 ambari-logsearch/ambari-logsearch-web/src/main/webapp/templates/troubleshoot/TroubleShootLayoutView_tmpl.html f72a7dd ambari-logsearch/ambari-logsearch-web/src/main/webapp/themejs/1.3.0/bootstrap.min.js 3807407 ambari-logsearch/docker/bin/start.sh c4ee06c ambari-logsearch/docker/logsearch-docker.sh 9d73eae Diff: https://reviews.apache.org/r/62034/diff/3/ Changes: https://reviews.apache.org/r/62034/diff/2-3/ Testing --- done. Thanks, Oliver Szabo
Re: Review Request 62034: Integrate LogSearch new UI with the server and get rid of the old one
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62034/ --- (Updated Sept. 1, 2017, 2:56 p.m.) Review request for Ambari, Andrii Babiichuk, Miklos Gergely, and Robert Nettleton. Changes --- make it work with spring-boot:run maven task Bugs: AMBARI-21870 https://issues.apache.org/jira/browse/AMBARI-21870 Repository: ambari Description --- - get rid of old ui - renamve web-new to web - add some small changes to authentication (create an abstract filter for jwt and move it somewhere else, those changes will be continued in an other patch) - change login/logout authentication mechanism for the new UI (do not redirect to /login.html) Diffs (updated) - ambari-logsearch/ambari-logsearch-server/build.xml aadacd7 ambari-logsearch/ambari-logsearch-server/pom.xml e60cc14 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/LogSearch.java 39343c7 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/auth/filter/AbstractJWTFilter.java PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/auth/model/JWTAuthenticationToken.java PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SecurityConfig.java 038ba5c ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/authenticate/LogsearchLogoutSuccessHandler.java c20e383 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/filters/LogsearchJWTFilter.java 1bc7231 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/model/JWTAuthenticationToken.java dfac191 ambari-logsearch/ambari-logsearch-web-new/.angular-cli.json cc74739 ambari-logsearch/ambari-logsearch-web-new/.editorconfig 6e87a00 ambari-logsearch/ambari-logsearch-web-new/.gitignore 54bfd20 ambari-logsearch/ambari-logsearch-web-new/README.md 2fb07fb ambari-logsearch/ambari-logsearch-web-new/e2e/app.e2e-spec.ts ebfdcb9 ambari-logsearch/ambari-logsearch-web-new/e2e/app.po.ts c446bfb ambari-logsearch/ambari-logsearch-web-new/e2e/tsconfig.e2e.json ac7a373 ambari-logsearch/ambari-logsearch-web-new/karma.conf.js 0512808 ambari-logsearch/ambari-logsearch-web-new/package.json 96733eb ambari-logsearch/ambari-logsearch-web-new/pom.xml 28e1754 ambari-logsearch/ambari-logsearch-web-new/protractor.conf.js 6b4a31c ambari-logsearch/ambari-logsearch-web-new/src/app/app.module.ts c08cb3a ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/audit-logs-query-params.class.ts ef8cf28 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/query-params.class.ts 83c3261 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-histogram-query-params.class.spec.ts efa2459 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-histogram-query-params.class.ts 87e82f6 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-query-params.class.ts c88ec46 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.html 1b134c3 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.less 8366952 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.spec.ts 3706d51 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.ts 131edcd ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.html a0444c9 ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.less d1aa7ff ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.spec.ts bc16ea4 ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.ts 4de47ea ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.html a16b205 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.less 55699b4 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.spec.ts 8efe320 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.ts 5800190 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.html 1baebed ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.less d47160f ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.spec.ts 5409d30
Review Request 62034: Integrate LogSearch new UI with the server and get rid of the old one
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62034/ --- Review request for Ambari, Andrii Babiichuk, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-21870 https://issues.apache.org/jira/browse/AMBARI-21870 Repository: ambari Description --- - get rid of old ui - renamve web-new to web - add some small changes to authentication (create an abstract filter for jwt and move it somewhere else, those changes will be continued in an other patch) - change login/logout authentication mechanism for the new UI (do not redirect to /login.html) Diffs - ambari-logsearch/ambari-logsearch-server/build.xml aadacd7 ambari-logsearch/ambari-logsearch-server/pom.xml e60cc14 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/LogSearch.java 39343c7 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/auth/filter/AbstractJWTFilter.java PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/auth/model/JWTAuthenticationToken.java PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SecurityConfig.java 038ba5c ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/authenticate/LogsearchLogoutSuccessHandler.java c20e383 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/filters/LogsearchJWTFilter.java 1bc7231 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/model/JWTAuthenticationToken.java dfac191 ambari-logsearch/ambari-logsearch-web-new/.angular-cli.json cc74739 ambari-logsearch/ambari-logsearch-web-new/.editorconfig 6e87a00 ambari-logsearch/ambari-logsearch-web-new/.gitignore 54bfd20 ambari-logsearch/ambari-logsearch-web-new/README.md 2fb07fb ambari-logsearch/ambari-logsearch-web-new/e2e/app.e2e-spec.ts ebfdcb9 ambari-logsearch/ambari-logsearch-web-new/e2e/app.po.ts c446bfb ambari-logsearch/ambari-logsearch-web-new/e2e/tsconfig.e2e.json ac7a373 ambari-logsearch/ambari-logsearch-web-new/karma.conf.js 0512808 ambari-logsearch/ambari-logsearch-web-new/package.json 96733eb ambari-logsearch/ambari-logsearch-web-new/pom.xml 28e1754 ambari-logsearch/ambari-logsearch-web-new/protractor.conf.js 6b4a31c ambari-logsearch/ambari-logsearch-web-new/src/app/app.module.ts c08cb3a ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/audit-logs-query-params.class.ts ef8cf28 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/query-params.class.ts 83c3261 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-histogram-query-params.class.spec.ts efa2459 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-histogram-query-params.class.ts 87e82f6 ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-query-params.class.ts c88ec46 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.html 1b134c3 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.less 8366952 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.spec.ts 3706d51 ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.ts 131edcd ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.html a0444c9 ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.less d1aa7ff ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.spec.ts bc16ea4 ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.ts 4de47ea ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.html a16b205 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.less 55699b4 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.spec.ts 8efe320 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.ts 5800190 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.html 1baebed ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.less d47160f ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.spec.ts 5409d30 ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.ts 3de664e
Re: Review Request 62015: Ranger should handle external Solr properly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62015/#review184272 --- Ship it! Ship It! - Oliver Szabo On Aug. 31, 2017, 1:59 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62015/ > --- > > (Updated Aug. 31, 2017, 1:59 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21863 > https://issues.apache.org/jira/browse/AMBARI-21863 > > > Repository: ambari > > > Description > --- > > Ranger tries to create it's configurations using the local ambari-infra Solr > even if it is specified to use an external one. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py > bd45558 > > > Diff: https://reviews.apache.org/r/62015/diff/1/ > > > Testing > --- > > Tested on two clusters, one running the external solr, the other installed > Atlas, the config data were created. > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61993: Atlas should handle external Solr properly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61993/#review184188 --- I know originally i recommended that change for atlas (and ranger as well), but on atlas side, external kerberos is not really handled, so maybe we can keep this open as it can cause complications on secured environment, I would say focus on the ranger change and keep this one open. - Oliver Szabo On Aug. 30, 2017, 4:04 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61993/ > --- > > (Updated Aug. 30, 2017, 4:04 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21855 > https://issues.apache.org/jira/browse/AMBARI-21855 > > > Repository: ambari > > > Description > --- > > Atlas tries to create it's collection using the local ambari-infra Solr even > if it is specified to use an external one. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py > 0305b82 > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py > 41be001 > > > Diff: https://reviews.apache.org/r/61993/diff/1/ > > > Testing > --- > > Tested on two clusters, one running the external solr, the other installed > Atlas, the collections were created. > > > Thanks, > > Miklos Gergely > >
Review Request 61996: Integrate new Log Search UI with maven
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61996/ --- Review request for Ambari, Andrii Babiichuk, Miklos Gergely, and Robert Nettleton. Bugs: AMBARI-21857 https://issues.apache.org/jira/browse/AMBARI-21857 Repository: ambari Description --- Add new pom.xml file for the new logsearch web ui in order to package static resources in jar (which are generated with yarn and angular-cli) The resource generation flow is the following: - install (download or copy an already downloaded one) node and yarn in target/webapp-build folder - copy source files from the ui root folder into target/webapp-build - run yarn install, then yarn build-prod - copy target/webapp-build/dist to target/classes/dist - during package phase target/classes content included in the generated jar Next steps from here: Add the new ui dependency to logsearch server, during the logsearch portal build, the dist static content will be packaged with the server files. Diffs - ambari-logsearch/ambari-logsearch-web-new/package.json b4902a3 ambari-logsearch/ambari-logsearch-web-new/pom.xml PRE-CREATION Diff: https://reviews.apache.org/r/61996/diff/1/ Testing --- done, manually, generated jar contains "dist/" folder. anyway, no mvn clean test support yet. Thanks, Oliver Szabo
Re: Review Request 61971: Log Search UI GA - merge into trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61971/#review184073 --- Ship it! Ship It! - Oliver Szabo On Aug. 29, 2017, 4:29 p.m., Andrii Babiichuk wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61971/ > --- > > (Updated Aug. 29, 2017, 4:29 p.m.) > > > Review request for Ambari and Andrii Tkach. > > > Bugs: AMBARI-21308 > https://issues.apache.org/jira/browse/AMBARI-21308 > > > Repository: ambari > > > Description > --- > > Merge nwe Log Search UI into trunk (to be located in seperate directory) > > > Diffs > - > > ambari-logsearch/ambari-logsearch-web-new/.angular-cli.json PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/.editorconfig PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/.gitignore PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/README.md PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/e2e/app.e2e-spec.ts PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/e2e/app.po.ts PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/e2e/tsconfig.e2e.json > PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/karma.conf.js PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/package.json PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/protractor.conf.js PRE-CREATION > ambari-logsearch/ambari-logsearch-web-new/src/app/app.module.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/audit-logs-query-params.class.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/query-params.class.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-histogram-query-params.class.spec.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-histogram-query-params.class.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/classes/queries/service-logs-query-params.class.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.html > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.less > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.spec.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/accordion-panel/accordion-panel.component.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.html > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.less > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.spec.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/app.component.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.html > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.less > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.spec.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-button/dropdown-button.component.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.html > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.less > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.spec.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/dropdown-list/dropdown-list.component.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/filter-button/filter-button.component.spec.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-new/src/app/components/filter-button/filter-button.component.ts > PRE-CREATION > > ambari-logsearch/ambari-logsearch-web-ne
Re: Review Request 61937: Fix new logsearch UI rat issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61937/ --- (Updated Aug. 28, 2017, 12:46 p.m.) Review request for Ambari, Andrii Babiichuk and Miklos Gergely. Changes --- add trunk as it will be merged later Bugs: AMBARI-21825 https://issues.apache.org/jira/browse/AMBARI-21825 Repository: ambari Description --- exclude some files from rat checks: vendor files, .* and README.md files (note: by the way, i do not know the .editorconfig file is required for us or not) Diffs - ambari-logsearch/pom.xml 2be11ee Diff: https://reviews.apache.org/r/61937/diff/2/ Testing --- ran mvn clean package successfully Thanks, Oliver Szabo
Re: Review Request 61937: Fix new logsearch UI rat issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61937/ --- (Updated Aug. 28, 2017, 11:58 a.m.) Review request for Ambari, Andrii Babiichuk and Miklos Gergely. Changes --- add recommended changes Bugs: AMBARI-21825 https://issues.apache.org/jira/browse/AMBARI-21825 Repository: ambari Description --- exclude some files from rat checks: vendor files, .* and README.md files (note: by the way, i do not know the .editorconfig file is required for us or not) Diffs (updated) - ambari-logsearch/pom.xml 2be11ee Diff: https://reviews.apache.org/r/61937/diff/2/ Changes: https://reviews.apache.org/r/61937/diff/1-2/ Testing --- ran mvn clean package successfully Thanks, Oliver Szabo
Review Request 61937: Fix new logsearch UI rat issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61937/ --- Review request for Ambari, Andrii Babiichuk and Miklos Gergely. Bugs: AMBARI-21825 https://issues.apache.org/jira/browse/AMBARI-21825 Repository: ambari Description --- exclude some files from rat checks: vendor files, .* and README.md files (note: by the way, i do not know the .editorconfig file is required for us or not) Diffs - ambari-logsearch/ambari-logsearch-web-new/README.md 2fb07fb ambari-logsearch/pom.xml 2be11ee Diff: https://reviews.apache.org/r/61937/diff/1/ Testing --- ran mvn clean package successfully Thanks, Oliver Szabo
Re: Review Request 61898: Create Utility Script to support Solr Collection Data Retention/Purging/Archiving
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61898/#review183863 --- ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java Lines 30 (patched) <https://reviews.apache.org/r/61898/#comment259927> rename the script name here as well (or remove it) - Oliver Szabo On Aug. 25, 2017, 4:20 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61898/ > --- > > (Updated Aug. 25, 2017, 4:20 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21810 > https://issues.apache.org/jira/browse/AMBARI-21810 > > > Repository: ambari > > > Description > --- > > This script allows the user to set an end date, and save all the previous log > entries into HDFS or S3 > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/build.xml 9b8b6cc > ambari-infra/ambari-infra-solr-client/pom.xml 3818aba > > ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java > PRE-CREATION > ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61898/diff/2/ > > > Testing > --- > > Tested on local cluster with > - SSL > - kerberos > - interrupted, then restarted script > - HDFS > - S3 > - the combinations of the above > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61898: Create Utility Script to support Solr Collection Data Retention/Purging/Archiving
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61898/#review183860 --- Ship it! Ship It! - Oliver Szabo On Aug. 25, 2017, 4:20 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61898/ > --- > > (Updated Aug. 25, 2017, 4:20 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21810 > https://issues.apache.org/jira/browse/AMBARI-21810 > > > Repository: ambari > > > Description > --- > > This script allows the user to set an end date, and save all the previous log > entries into HDFS or S3 > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/build.xml 9b8b6cc > ambari-infra/ambari-infra-solr-client/pom.xml 3818aba > > ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java > PRE-CREATION > ambari-infra/ambari-infra-solr-client/src/main/python/solrDataManager.py > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61898/diff/2/ > > > Testing > --- > > Tested on local cluster with > - SSL > - kerberos > - interrupted, then restarted script > - HDFS > - S3 > - the combinations of the above > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61898: Create Utility Script to support Solr Collection Data Retention/Purging/Archiving
> On Aug. 25, 2017, 9:55 a.m., Oliver Szabo wrote: > > ambari-infra/ambari-infra-solr-client/src/main/python/solr_collection_data_manager.py > > Lines 1 (patched) > > <https://reviews.apache.org/r/61898/diff/1/?file=1803361#file1803361line1> > > > > i think we do not need the solr_collection prefix for the script name, > > it can be only data_manager or data_archiver (if we could only delete > > records without archiving, data_manager is the correct one, otherwise > > data_archiver can be enogh) maybe the name should look like solrDataManager.py as the shell script names looks like that, so solrDataArchiver.py or solrDataManager.py would be better - Oliver --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61898/#review183838 --- On Aug. 24, 2017, 11:42 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61898/ > --- > > (Updated Aug. 24, 2017, 11:42 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21810 > https://issues.apache.org/jira/browse/AMBARI-21810 > > > Repository: ambari > > > Description > --- > > This script allows the user to set an end date, and save all the previous log > entries into HDFS or S3 > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/build.xml 9b8b6cc > ambari-infra/ambari-infra-solr-client/pom.xml 3818aba > > ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java > PRE-CREATION > > ambari-infra/ambari-infra-solr-client/src/main/python/solr_collection_data_manager.py > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61898/diff/1/ > > > Testing > --- > > Tested on local cluster with > - SSL > - kerberos > - interrupted, then restarted script > - HDFS > - S3 > - the combinations of the above > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61898: Create Utility Script to support Solr Collection Data Retention/Purging/Archiving
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61898/#review183838 --- ambari-infra/ambari-infra-solr-client/src/main/python/solr_collection_data_manager.py Lines 1 (patched) <https://reviews.apache.org/r/61898/#comment259907> i think we do not need the solr_collection prefix for the script name, it can be only data_manager or data_archiver (if we could only delete records without archiving, data_manager is the correct one, otherwise data_archiver can be enogh) - Oliver Szabo On Aug. 24, 2017, 11:42 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61898/ > --- > > (Updated Aug. 24, 2017, 11:42 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21810 > https://issues.apache.org/jira/browse/AMBARI-21810 > > > Repository: ambari > > > Description > --- > > This script allows the user to set an end date, and save all the previous log > entries into HDFS or S3 > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/build.xml 9b8b6cc > ambari-infra/ambari-infra-solr-client/pom.xml 3818aba > > ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java > PRE-CREATION > > ambari-infra/ambari-infra-solr-client/src/main/python/solr_collection_data_manager.py > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61898/diff/1/ > > > Testing > --- > > Tested on local cluster with > - SSL > - kerberos > - interrupted, then restarted script > - HDFS > - S3 > - the combinations of the above > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61898: Create Utility Script to support Solr Collection Data Retention/Purging/Archiving
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61898/#review183810 --- ambari-infra/ambari-infra-solr-client/src/main/python/solr_collection_data_manager.py Lines 286 (patched) <https://reviews.apache.org/r/61898/#comment259859> remove comment ambari-infra/ambari-infra-solr-client/src/main/python/solr_collection_data_manager.py Lines 288 (patched) <https://reviews.apache.org/r/61898/#comment259860> remove comment - Oliver Szabo On Aug. 24, 2017, 11:42 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61898/ > --- > > (Updated Aug. 24, 2017, 11:42 p.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21810 > https://issues.apache.org/jira/browse/AMBARI-21810 > > > Repository: ambari > > > Description > --- > > This script allows the user to set an end date, and save all the previous log > entries into HDFS or S3 > > > Diffs > - > > ambari-infra/ambari-infra-solr-client/build.xml 9b8b6cc > ambari-infra/ambari-infra-solr-client/pom.xml 3818aba > > ambari-infra/ambari-infra-solr-client/src/main/java/org/apache/ambari/infra/solr/S3Uploader.java > PRE-CREATION > > ambari-infra/ambari-infra-solr-client/src/main/python/solr_collection_data_manager.py > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61898/diff/1/ > > > Testing > --- > > Tested on local cluster with > - SSL > - kerberos > - interrupted, then restarted script > - HDFS > - S3 > - the combinations of the above > > > Thanks, > > Miklos Gergely > >
