> On Aug. 24, 2016, 10:17 a.m., Balázs Donát Bessenyei wrote: > > flume-ng-doc/sphinx/FlumeUserGuide.rst, lines 246-247 > > <https://reviews.apache.org/r/51182/diff/4/?file=1483597#file1483597line246> > > > > nit: still a few spelling errors
fixed > On Aug. 24, 2016, 10:17 a.m., Balázs Donát Bessenyei wrote: > > flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java, > > line 180 > > <https://reviews.apache.org/r/51182/diff/4/?file=1483600#file1483600line180> > > > > With LogRawDataUtil (raw data logging enabled) this could be logged added LogRawDataUtil > On Aug. 24, 2016, 10:17 a.m., Balázs Donát Bessenyei wrote: > > flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java, > > line 305 > > <https://reviews.apache.org/r/51182/diff/4/?file=1483600#file1483600line305> > > > > With using LogRawDataUtil (configuration setting) I guess, this could > > be logged. added LogRawDataUtil. since properties for KafkaProducer is dynamically built (flume adds, renames and alters them during startup) it is good to be logged to see the modifications comparable to the properties logged at startup. > On Aug. 24, 2016, 10:17 a.m., Balázs Donát Bessenyei wrote: > > flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java, > > line 374 > > <https://reviews.apache.org/r/51182/diff/4/?file=1483600#file1483600line374> > > > > I guess we should use the LogRawDataUtil here this happens right before KafkaSink#304 and does the same: "logs out the kafkaProps field". So removing this line eliminates a duplicate. > On Aug. 24, 2016, 10:17 a.m., Balázs Donát Bessenyei wrote: > > flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/MorphlineSink.java, > > line 142 > > <https://reviews.apache.org/r/51182/diff/4/?file=1483602#file1483602line142> > > > > {}, event seems to be missing event was added > On Aug. 24, 2016, 10:17 a.m., Balázs Donát Bessenyei wrote: > > flume-ng-sources/flume-twitter-source/src/main/java/org/apache/flume/source/twitter/TwitterSource.java, > > lines 110-113 > > <https://reviews.apache.org/r/51182/diff/4/?file=1483604#file1483604line110> > > > > With LogRawDataUtil config logging, this could be logged all properties for the agent are logged if LogRawDataUtil.LOG_PRINTCONFIG was set at startup including these above 2016-08-24 13:44:06,803 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:316)] Initial-configuration: AgentConfiguration[a1] SOURCES: {r1={ parameters:{port=44444, channels=c1, type=netcat, bind=localhost} }} CHANNELS: {c1={ parameters:{transactionCapacity=100, capacity=1000, type=memory} }} SINKS: {kafka={ parameters:{kafka.doesntexists=1} }, k1={ parameters:{type=logger, channel=c1} }} so logging them here again is a clear duplicate. - Attila ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51182/#review146634 ----------------------------------------------------------- On Aug. 24, 2016, 11:58 a.m., Attila Simon wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51182/ > ----------------------------------------------------------- > > (Updated Aug. 24, 2016, 11:58 a.m.) > > > Review request for Flume. > > > Bugs: FLUME-2954 > https://issues.apache.org/jira/browse/FLUME-2954 > > > Repository: flume-git > > > Description > ------- > > -------------------------------------------------------------------------------- > flume-ng-channel --- > flume-jdbc-channel --- > JdbcChannelProviderImpl#98 <- fail properties <REMOVED> > JdbcChannelProviderImpl#261 #431 <- fail properties: jdbc url > might include password <KEPT><FOLLOWUP IN JIRA> > flume-kafka-channel --- > KafkaChannel#230 #253 <- fail properties <REMOVED> > -------------------------------------------------------------------------------- > flume-ng-configuration --- > FlumeConfiguration#315 #372 <- fail properties <DRIVE BY > PROPERTY> > -------------------------------------------------------------------------------- > flume-ng-core --- > SyslogAvroEventSerializer#150 <- fail data: > SyslogEvent.message gets logged <DRIVE BY PROPERTY> > GangliaServer#224 #245 <- safe data: only flume > component metrics data <KEPT> > LoggerSink#95 <- fail data: on purpose <KEPT> > AvroSource#347 <- fail data: log whole message > <DRIVE BY PROPERTY> > MultiportSyslogTCPSource#360 <- fail data: log whole message > <DRIVE BY PROPERTY> > BLOBHandler#70 <- fail data: logs http request > headers <DRIVE BY PROPERTY> > -------------------------------------------------------------------q------------- > flume-ng-embedded-agent --- > EmbeddedAgent#155 <- fail properties: printing > all config <DRIVE BY PROPERTY> > -------------------------------------------------------------------------------- > flume-ng-sinks --- > flume-hive-sink --- > HiveEndPoint has an URI field. <- fail properties > <KEPT><FOLLOWUP IN JIRA> > It may contain private data > (URI string may contain password) as it is > excessively logged within this module. > Appears in HiveSink#298 #342 #400 #403 #428, > HiveWriter#210 #319 #330 #337 #353 #365 #368 #407...) > HiveEndPoint is also attached to exception logs as well > flume-ng-hbase-sink --- > AsyncHBaseSink#641 <- safe data: error details > gets logged in case of failure <KEPT> > flume-ng-kafka-sink --- > KafkaSink#179 <- fail data: log whole message > <REMOVED> > KafkaSink#304 <- fail properties <REMOVED> > flume-ng-morphline-solr-sink --- > BlobHandler#98 #113 <- fail data: log http request > headers <DRIVE BY PROPERTY> > MorphlineSink#139 <- fail data: logs event <DRIVE > BY PROPERTY> > -------------------------------------------------------------------------------- > flume-ng-sources --- > flume-kafka-source --- > KafkaSource#247 <- fail data: log whole <DRIVE > BY PROPERTY> > flume-twitter-source --- > TwitterSource#110-113 <- fail properties <REMOVED> > -------------------------------------------------------------------------------- > > > Diffs > ----- > > conf/flume-env.ps1.template 8bf535a > conf/flume-env.sh.template c8b660f > > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java > 845b794 > > flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java > 90e3288 > > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java > 9b3a434 > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 8b9b956 > > flume-ng-core/src/main/java/org/apache/flume/source/MultiportSyslogTCPSource.java > b9f2438 > flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java > e24d4c6 > > flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java > 05af3b1 > flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6 > > flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java > ad3e138 > flume-ng-sdk/src/main/java/org/apache/flume/util/LogRawDataUtil.java > PRE-CREATION > > flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java > 9453546 > > flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java > ca7614a > > flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/MorphlineSink.java > f7a73f3 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > 90e4715 > > flume-ng-sources/flume-twitter-source/src/main/java/org/apache/flume/source/twitter/TwitterSource.java > f5c8328 > > Diff: https://reviews.apache.org/r/51182/diff/ > > > Testing > ------- > > compiles, site builds, all unit test passes, distribution target handles the > system properties as expected: > bin/flume-ng agent --conf conf --conf-file > ../../../../../flume-conf/flume-log.conf --name a1 > -Dflume.root.logger=DEBUG,console -Dorg.apache.flume.log.printconfig=true > -Dorg.apache.flume.log.rawdata=true (with and without the extra properties) > > > Thanks, > > Attila Simon > >
